grin server crash: `thread 'main' panicked at 'already mutably borrowed: BorrowError'`

[kingsley-shacklebolt]

I compiled the server and joined the testnet to try it out. I was running code compiled from commit df34fe6, on the milestone/testnet1 branch. I just checked my terminal and noticed that the server had crashed. Here is the output:

$ grin server -m run
Dec 10 05:11:05.737 WARN P2P server started on 0.0.0.0:13414
Dec 10 05:11:05.738 WARN Grin server started.
thread 'main' panicked at 'already mutably borrowed: BorrowError', src/libcore/result.rs:906:4
note: Run with `RUST_BACKTRACE=1` for a backtrace.

The last log timestamp from grin wallet -e listen was at Dec 10 16:07:29.326, so I believe the server crashed around 11 hours after it was started.

I believe this is the line from libcore: https://github.com/rust-lang/rust/blob/master/src/libcore/result.rs#L906

I've just pulled the latest commit (cccb558), and I've started the server again with $ RUST_BACKTRACE=1 grin server -m run. I'll be able to provide a full backtrace if it ever crashes again. (Maybe one of the 3 most recent commits fixed the issue? I couldn't tell.)

[sesam]

Thanks for reporting! The culprit libcore/result.rs around 906 is familiar https://github.com/mimblewimble/grin/issues?utf8=%E2%9C%93&q=is%3Aissue+BorrowError

I had thought it was fixed. Twice closed (#400, #421) and 3 still open issues (#430, #453, and this #456 that I'm now commenting on)

I might have seen this myself, but probably just restarted and got past the problem.

Do you @kingsley-shacklebolt get this error repeatedly, or only once?

If it repeats, try running RUST_BACKTRACE=1 grin ... for both the wallet and the server, and see if you get a more detailed backtrace. Line 906 is the Ok option, not the default, and we need to find the callee -- so should be visible in a backtrace.

[sesam]

The previous two open #430 and #453 both have this in their backtrace:

11: 0x55d8728d1979 - <core::cell::RefCell>::borrow::hbd865b827be982af at /checkout/src/libcore/cell.rs:669 12: 0x55d8728cedfe - <grin_util::OneTime>::borrow::h44ce1c23b1eadfb0 at /root/grin-flip/util/src/lib.rs:98 13: 0x55d872aad917 - ::peer_addrs_received::hbb92f6ae27862c90 at grin/src/adapters.rs:212

so it would be nice if you get the same, then we might have a single bug to squash for 3 issues worth :)

[kingsley-shacklebolt]

Hi @sesam, yes I've just updated to the latest commit on milestone/testnet1, and have started the server with RUST_BACKTRACE=1. I'll also use that with the wallet.

I just realized that I should also delete everything and start syncing from scratch, because now I'm all caught up and the server isn't doing anything. So I'll do that now, and report back if the crash happens again.

[kingsley-shacklebolt]

Just letting you know that I synced up to block 21710 with no crashes. I'll do it a few more times and see if I can reproduce the issue.

[hendi]

I've catched a similar one:

Dec 11 12:08:43.372 WARN P2P server started on 0.0.0.0:13414
Dec 11 12:08:43.373 WARN Grin server started.
thread 'main' panicked at 'already mutably borrowed: BorrowError', /checkout/src/libcore/result.rs:860:4
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
             at /checkout/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at /checkout/src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at /checkout/src/libstd/sys_common/backtrace.rs:60
             at /checkout/src/libstd/panicking.rs:380
   3: std::panicking::default_hook
             at /checkout/src/libstd/panicking.rs:396
   4: std::panicking::rust_panic_with_hook
             at /checkout/src/libstd/panicking.rs:611
   5: std::panicking::begin_panic_new
             at /checkout/src/libstd/panicking.rs:553
   6: std::panicking::begin_panic_fmt
             at /checkout/src/libstd/panicking.rs:521
   7: rust_begin_unwind
             at /checkout/src/libstd/panicking.rs:497
   8: core::panicking::panic_fmt
             at /checkout/src/libcore/panicking.rs:92
   9: core::result::unwrap_failed
             at /checkout/src/libcore/macros.rs:31
  10: <core::result::Result<T, E>>::expect
             at /checkout/src/libcore/result.rs:762
  11: <core::cell::RefCell<T>>::borrow
             at /checkout/src/libcore/cell.rs:588
  12: <grin_util::OneTime<T>>::borrow
             at /home/hendi/Projects/grin/util/src/lib.rs:98
  13: <grin_grin::adapters::NetToChainAdapter as grin_p2p::types::NetAdapter>::peer_addrs_received
             at grin/src/adapters.rs:212
  14: <grin_p2p::peer::TrackingAdapter as grin_p2p::types::NetAdapter>::peer_addrs_received
             at p2p/src/peer.rs:260
  15: grin_p2p::protocol::handle_payload
             at p2p/src/protocol.rs:247
  16: <grin_p2p::protocol::ProtocolV1 as grin_p2p::types::Protocol>::handle::{{closure}}
             at p2p/src/protocol.rs:55
  17: <F as grin_p2p::conn::Handler>::handle
             at p2p/src/conn.rs:66
  18: grin_p2p::conn::TimeoutConnection::listen::{{closure}}
             at p2p/src/conn.rs:275
  19: <F as grin_p2p::conn::Handler>::handle
             at p2p/src/conn.rs:66
  20: grin_p2p::conn::Connection::read_msg::{{closure}}::{{closure}}
             at p2p/src/conn.rs:215
  21: <futures::future::and_then::AndThen<A, B, F> as futures::future::Future>::poll::{{closure}}::{{closure}}
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/and_then.rs:34
  22: <core::result::Result<T, E>>::map
             at /checkout/src/libcore/result.rs:458
  23: <futures::future::and_then::AndThen<A, B, F> as futures::future::Future>::poll::{{closure}}
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/and_then.rs:33
  24: <futures::future::chain::Chain<A, B, C>>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/chain.rs:39
  25: <futures::future::and_then::AndThen<A, B, F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/and_then.rs:32
  26: <futures::stream::fold::Fold<S, F, Fut, T> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/stream/fold.rs:70
  27: <alloc::boxed::Box<F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/mod.rs:113
  28: <futures::future::map::Map<A, F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/map.rs:30
  29: <futures::future::select::Select<A, B> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/select.rs:48
  30: <futures::future::map::Map<A, F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/map.rs:30
  31: <futures::future::map_err::MapErr<A, F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/map_err.rs:30
  32: <futures::future::select::Select<A, B> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/select.rs:52
  33: <futures::future::map::Map<A, F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/map.rs:30
  34: <futures::future::map_err::MapErr<A, F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/map_err.rs:30
  35: <alloc::boxed::Box<F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/mod.rs:113
  36: <futures::future::select::Select<A, B> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/select.rs:48
  37: <futures::future::map::Map<A, F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/map.rs:30
  38: <futures::future::map_err::MapErr<A, F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/map_err.rs:30
  39: <alloc::boxed::Box<F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/mod.rs:113
  40: <futures::future::chain::Chain<A, B, C>>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/chain.rs:26
  41: <futures::future::then::Then<A, B, F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/then.rs:32
  42: <alloc::boxed::Box<F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/mod.rs:113
  43: <futures::future::map_err::MapErr<A, F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/map_err.rs:30
  44: <alloc::boxed::Box<F> as futures::future::Future>::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/mod.rs:113
  45: <futures::task_impl::Spawn<T>>::poll_future_notify::{{closure}}
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/task_impl/mod.rs:288
  46: <futures::task_impl::Spawn<T>>::enter::{{closure}}
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/task_impl/mod.rs:362
  47: futures::task_impl::std::set
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/task_impl/std/mod.rs:78
  48: <futures::task_impl::Spawn<T>>::enter
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/task_impl/mod.rs:362
  49: <futures::task_impl::Spawn<T>>::poll_future_notify
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/task_impl/mod.rs:288
  50: tokio_core::reactor::Core::dispatch_task::{{closure}}
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.10/src/reactor/mod.rs:356
  51: <scoped_tls::ScopedKey<T>>::set
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/scoped-tls-0.1.0/src/lib.rs:135
  52: tokio_core::reactor::Core::dispatch_task
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.10/src/reactor/mod.rs:355
  53: tokio_core::reactor::Core::dispatch
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.10/src/reactor/mod.rs:316
  54: tokio_core::reactor::Core::poll
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.10/src/reactor/mod.rs:304
  55: tokio_core::reactor::Core::run
             at /home/hendi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.10/src/reactor/mod.rs:241
  56: grin_grin::server::Server::start
             at grin/src/server.rs:76
  57: grin::server_command
             at src/bin/grin.rs:341
  58: grin::main
             at src/bin/grin.rs:270
  59: __rust_maybe_catch_panic
             at /checkout/src/libpanic_unwind/lib.rs:98
  60: std::rt::lang_start
             at /checkout/src/libstd/panicking.rs:458
             at /checkout/src/libstd/panic.rs:361
             at /checkout/src/libstd/rt.rs:59
  61: main
  62: __libc_start_main
  63: <unknown>```