Double panic when trying to run `cargo test` on Win10

[ErichDonGubler]

For some reason tests inside of src/dynamic_honey_badger/votes.rs are throwing panics inside of panics. This causes an immediate abort, and blocks tests from running. This seems specific to my Windows 10 machine, since master seems to pass just fine in Travis (a *nix environment).

Output of cargo test:

C:\Users\egubler\workspace\personal\pull-requests\hbbft>cargo test
    Finished dev [unoptimized + debuginfo] target(s) in 0.53s
     Running target\debug\deps\hbbft-0cddedd180c2030e.exe

running 3 tests
test broadcast::merkle::tests::test_merkle ... ok
test dynamic_honey_badger::votes::tests::test_pending_votes ... FAILED
thread panicked while panicking. aborting.
error: test failed, to rerun pass '--lib'

Output of rustup show:

active toolchain
----------------

stable-x86_64-pc-windows-gnu (default)
rustc 1.29.0 (aa3ca1994 2018-09-11)

**Output of `RUST_BACKTRACE=1 cargo test` in MSYS2 environment**

``` ~/…/pull-requests/hbbft master$ RUST_BACKTRACE=1 cargo test Finished dev [unoptimized + debuginfo] target(s) in 0.57s Running target\debug\deps\hbbft-0cddedd180c2030e.exe running 3 tests test broadcast::merkle::tests::test_merkle ... ok test dynamic_honey_badger::votes::tests::test_committed_votes ... FAILED test dynamic_honey_badger::votes::tests::test_pending_votes ... FAILED failures: ---- dynamic_honey_badger::votes::tests::test_committed_votes stdout ---- thread 'dynamic_honey_badger::votes::tests::test_committed_votes' panicked at 'Failed to drop `SecretKey`: Failed to `munlock` 32 bytes starting at address: 0x1a90c30', C:\Users\egubler\.cargo\git\checkouts\threshold_crypto-3605c1a1492f376b\02109b5\src\lib.rs:308:13 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. stack backtrace: 0: std::sys::windows::backtrace::unwind_backtrace at libstd\sys\windows\backtrace/mod.rs:104 at libstd\sys\windows\backtrace/mod.rs:91 1: std::sys_common::backtrace::print at libstd\sys_common/backtrace.rs:71 at libstd\sys_common/backtrace.rs:59 2: std::panicking::default_hook::{{closure}} at libstd/panicking.rs:211 3: std::panicking::default_hook at libstd/panicking.rs:221 4: std::panicking::rust_panic_with_hook at libstd/panicking.rs:475 5: std::panicking::continue_panic_fmt at libstd/panicking.rs:390 6: std::panicking::begin_panic_fmt at libstd/panicking.rs:345 7: ::drop at C:\Users\egubler\.cargo\git\checkouts\threshold_crypto-3605c1a1492f376b\02109b5\src/lib.rs:308 8: core::ptr::drop_in_place at C:\projects\rust\src\libcore/ptr.rs:59 9: core::ptr::drop_in_place at C:\projects\rust\src\libcore/ptr.rs:59 10: core::mem::drop at C:\projects\rust\src\libcore/mem.rs:795 11: core::ops::function::FnMut::call_mut at C:\projects\rust\src\libcore\ops/function.rs:146 12: core::iter::iterator::Iterator::for_each::{{closure}} at C:\projects\rust\src\libcore\iter/iterator.rs:553 13: core::iter::iterator::Iterator::fold::{{closure}} at C:\projects\rust\src\libcore\iter/iterator.rs:1634 14: core::iter::iterator::Iterator::try_fold at C:\projects\rust\src\libcore\iter/iterator.rs:1522 15: core::iter::iterator::Iterator::fold at C:\projects\rust\src\libcore\iter/iterator.rs:1634 16: core::iter::iterator::Iterator::for_each at C:\projects\rust\src\libcore\iter/iterator.rs:553 17: as core::ops::drop::Drop>::drop at C:\projects\rust\src\liballoc\collections\btree/map.rs:1313 18: core::ptr::drop_in_place at C:\projects\rust\src\libcore/ptr.rs:59 19: core::mem::drop at C:\projects\rust\src\libcore/mem.rs:795 20: as core::ops::drop::Drop>::drop at C:\projects\rust\src\liballoc\collections\btree/map.rs:144 21: core::ptr::drop_in_place at C:\projects\rust\src\libcore/ptr.rs:59 22: >::generate_map at src/messaging.rs:401 23: hbbft::dynamic_honey_badger::votes::tests::setup at src\dynamic_honey_badger/votes.rs:203 24: hbbft::dynamic_honey_badger::votes::tests::test_committed_votes at src\dynamic_honey_badger/votes.rs:279 25: hbbft::__test::TESTS::{{closure}} at src\dynamic_honey_badger/votes.rs:275 26: core::ops::function::FnOnce::call_once at C:\projects\rust\src\libcore\ops/function.rs:223 27: >::call_box at libtest/lib.rs:1451 at C:\projects\rust\src\libcore\ops/function.rs:223 at C:\projects\rust\src\liballoc/boxed.rs:642 28: _rust_maybe_catch_panic at libpanic_unwind/lib.rs:105 ---- dynamic_honey_badger::votes::tests::test_pending_votes stdout ---- thread 'dynamic_honey_badger::votes::tests::test_pending_votes' panicked at 'Failed to drop `SecretKey`: Failed to `munlock` 32 bytes starting at address: 0x1aa0620', C:\Users\egubler\.cargo\git\checkouts\threshold_crypto-3605c1a1492f376b\02109b5\src\lib.rs:308:13 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. stack backtrace: 0: std::sys::windows::backtrace::unwind_backtrace at libstd\sys\windows\backtrace/mod.rs:104 at libstd\sys\windows\backtrace/mod.rs:91 1: std::sys_common::backtrace::print at libstd\sys_common/backtrace.rs:71 at libstd\sys_common/backtrace.rs:59 2: std::panicking::default_hook::{{closure}} at libstd/panicking.rs:211 3: std::panicking::default_hook at libstd/panicking.rs:221 4: std::panicking::rust_panic_with_hook at libstd/panicking.rs:475 5: std::panicking::continue_panic_fmt at libstd/panicking.rs:390 6: std::panicking::begin_panic_fmt at libstd/panicking.rs:345 7: ::drop at C:\Users\egubler\.cargo\git\checkouts\threshold_crypto-3605c1a1492f376b\02109b5\src/lib.rs:308 8: core::ptr::drop_in_place at C:\projects\rust\src\libcore/ptr.rs:59 9: core::ptr::drop_in_place at C:\projects\rust\src\libcore/ptr.rs:59 10: core::mem::drop at C:\projects\rust\src\libcore/mem.rs:795 11: core::ops::function::FnMut::call_mut at C:\projects\rust\src\libcore\ops/function.rs:146 12: core::iter::iterator::Iterator::for_each::{{closure}} at C:\projects\rust\src\libcore\iter/iterator.rs:553 13: core::iter::iterator::Iterator::fold::{{closure}} at C:\projects\rust\src\libcore\iter/iterator.rs:1634 14: core::iter::iterator::Iterator::try_fold at C:\projects\rust\src\libcore\iter/iterator.rs:1522 15: core::iter::iterator::Iterator::fold at C:\projects\rust\src\libcore\iter/iterator.rs:1634 16: core::iter::iterator::Iterator::for_each at C:\projects\rust\src\libcore\iter/iterator.rs:553 17: as core::ops::drop::Drop>::drop at C:\projects\rust\src\liballoc\collections\btree/map.rs:1313 18: core::ptr::drop_in_place at C:\projects\rust\src\libcore/ptr.rs:59 19: core::mem::drop at C:\projects\rust\src\libcore/mem.rs:795 20: as core::ops::drop::Drop>::drop at C:\projects\rust\src\liballoc\collections\btree/map.rs:144 21: core::ptr::drop_in_place at C:\projects\rust\src\libcore/ptr.rs:59 22: >::generate_map at src/messaging.rs:401 23: hbbft::dynamic_honey_badger::votes::tests::setup at src\dynamic_honey_badger/votes.rs:203 24: hbbft::dynamic_honey_badger::votes::tests::test_pending_votes at src\dynamic_honey_badger/votes.rs:226 25: hbbft::__test::TESTS::{{closure}} at src\dynamic_honey_badger/votes.rs:222 26: core::ops::function::FnOnce::call_once at C:\projects\rust\src\libcore\ops/function.rs:223 27: >::call_box at libtest/lib.rs:1451 at C:\projects\rust\src\libcore\ops/function.rs:223 at C:\projects\rust\src\liballoc/boxed.rs:642 28: _rust_maybe_catch_panic at libpanic_unwind/lib.rs:105 failures: dynamic_honey_badger::votes::tests::test_committed_votes dynamic_honey_badger::votes::tests::test_pending_votes test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out error: test failed, to rerun pass '--lib' ```

If there's any other information I can provide to you, please let me know!

[ErichDonGubler]

<threshold_crypto::SecretKey as core::ops::drop::Drop>::drop in the stacktrace seems to be the culprit -- the panic message states 'Failed to drop `SecretKey`: Failed to `munlock` 32 bytes starting at address: 0x1aa0620. Is this a hole in Windows support for threshold_crypto?

[c0gent]

Yes. Unfortunately we haven't been testing on Windows at all. You shouldn't have any problems on MinGW (though no promises). Linux is our only first-class platform for the moment. This is something that badly needs to be addressed.

[ErichDonGubler]

@c0gent: In my report above I've noted that the target triple is stable-x86_64-pc-windows-gnu -- msvc doesn't even compile because unistd.h is missing. So, this problem is happening specifically in the MinGW environment.

[c0gent]

My mistake, I was on my phone and I missed that.

[vkomenda]

@ErichDonGubler, please retry with

MLOCK_SECRETS=false cargo test --release

as described in the README. Let know how it went.

[ErichDonGubler]

@vkomenda: MLOCK_SECRETS=false resolves my immediate issue (running tests), independent of the --release flag. I have not found a way to work around the issue as originally reported, but thank you for the tip!

[afck]

Does it work with test-threads=1? I wonder whether too many pages get locked at the same time.

Maybe we should consider making MLOCK_SECRETS=false the default until we have addressed this issue in general (see discussion in https://github.com/poanetwork/threshold_crypto/pull/34). I wonder whether it's worse on Windows (e.g. lower limit for locked pages, or a tendency to lock more pages because of different allocation).

[ErichDonGubler]

@afck: This is the error output I'm getting trying to use cargo test --release --test-threads 1:

$ cargo test --release -- --test-threads 1
   Compiling hbbft v0.1.0 (file:///C:/msys64/home/K0RYU/workspace/interviews/poa-networks/hbbft)
    Finished release [optimized] target(s) in 1m 42s
     Running target\release\deps\hbbft-efd7f2c9ff217d71.exe

running 3 tests
test broadcast::merkle::tests::test_merkle ... ok
test dynamic_honey_badger::votes::tests::test_committed_votes ... FAILED
test dynamic_honey_badger::votes::tests::test_pending_votes ... FAILED

failures:

---- dynamic_honey_badger::votes::tests::test_committed_votes stdout ----
thread 'dynamic_honey_badger::votes::tests::test_committed_votes' panicked at 'Failed to drop `SecretKey`: Failed to `munlock` 32 bytes starting at address: 0xbdd70', C:\Users\K0RYU\.cargo\git\checkouts\threshold_crypto-3605c1a1492f376b\e1c9158\src\lib.rs:316:13
note: Run with `RUST_BACKTRACE=1` for a backtrace.

---- dynamic_honey_badger::votes::tests::test_pending_votes stdout ----
thread 'dynamic_honey_badger::votes::tests::test_pending_votes' panicked at 'Failed to drop `SecretKey`: Failed to `munlock` 32 bytes starting at address: 0xbda10', C:\Users\K0RYU\.cargo\git\checkouts\threshold_crypto-3605c1a1492f376b\e1c9158\src\lib.rs:316:13

failures:
    dynamic_honey_badger::votes::tests::test_committed_votes
    dynamic_honey_badger::votes::tests::test_pending_votes

test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--lib'

[afck]

All mlock/munlock code has been removed from threshold_crypto for now. We have https://github.com/poanetwork/threshold_crypto/issues/24 and related issues to add it back in in the future, but this time with a dedicated allocator to avoid these kinds of problems.

(Please reopen if you can still reproduce the panic with current master.)