search

oasisprotocol / oasis-core

Performant and Confidentiality-Preserving Smart Contracts + Blockchains
https://oasisprotocol.org
Apache License 2.0
338 stars 112 forks source link

Playback benchmarks `HashMismatch` #1869

Closed ptrus closed 5 years ago

ptrus commented 5 years ago

Following happens on every run of the playback benchmark on the benchmarking cluster:

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: HashMismatch { expected_hash: 1693a54f44dafc6faa8a59530bf9fe8e27e875bd71018ec7ac9867fb1aa0ffc3, computed_hash: e1705f426eff0b47d4c73196457d1b4db5ebd8aa3349d329ac184699ada2e307 }

stack backtrace:
   0: failure::backtrace::internal::InternalBacktrace::new::h88fc859829218a81 (0x55d4fc99eb19)
   1: failure::backtrace::Backtrace::new::hdbcb3dafa05cdaac (0x55d4fc99ec19)', src/libcore/result.rs:999:5
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at ./cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.25/src/backtrace/libunwind.rs:97
   1: backtrace::backtrace::trace_unsynchronized
             at ./cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.25/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:47
   3: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:36
   4: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:197
   5: std::panicking::default_hook
             at src/libstd/panicking.rs:211
   6: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:474
   7: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:381
   8: rust_begin_unwind
             at src/libstd/panicking.rs:308
   9: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
  10: core::result::unwrap_failed
  11: ekiden_runtime::storage::mkvs::urkel::tree::mkvs::<impl ekiden_runtime::storage::mkvs::MKVS for ekiden_runtime::storage::mkvs::urkel::tree::tree::UrkelTree>::remove
  12: ekiden_runtime::storage::context::StorageContext::with_current
  13: <runtime_ethereum_common::storage::ThreadLocalMKVS as ethcore::mkvs::MKVS>::remove

Seems to consistently happen in the same "batch" of transactions.

Attached is a log of ~150 executed transactions that caused the panic on one of the runs:

playback-transactions.txt

Details

Steps to reproduce: run the playback benchmark? (tried reproducing this locally but didn't manage to get genesis state imported without running OOM every-time)

Not sure if there's an easy way to just insert transaction hashes in the DB and check if it can be reproduced without running the full benchmarks?

Acceptance Criteria

ptrus commented 5 years ago

dump: dump-full.txt

Logs:

{"msg":"Removing key","level":"WARN","ts":"2019-06-28T14:38:08.831579097+00:00","module":"whatever/whatever","key":"[203, 9, 39, 189, 199, 47, 241, 166, 155, 201, 198, 159, 207, 229, 87, 106, 183, 36, 253, 106, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2]"}
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: HashMismatch { expected_hash: 2ba65029cdb1ec2648d9bb949b11c16fa615be5aa5d9e94ed1ba6a2bf6b9641c, computed_hash: 34cd39eabc3b2adc1cabbf72616cbdd003e7f84142179220bae19a589007a6d7 }

edit: not sure if the dump is full? but that's what the dump function seems to write (the dump function completes)

ptrus commented 5 years ago

Cannot reproduce it without importing playback genesis state, but did manage to reproduce it by running a single transaction after the genesis state is imported:

Steps to reproduce locally:

1) Import playback genesis state:

rm -rf /tmp/ekiden-single-node; cp -R configs/single_node /tmp/ekiden-single-node && chmod -R go-rwx /tmp/ekiden-single-node && ./go/ekiden/ekiden --config configs/single_node.yml

./target/release/genesis-playback ./state-2999999.json roothash_blocks.json --node-address unix:/tmp/ekiden-single-node/internal.sock

2) Stop the node and generate new config with state roothash

Save storage:

cp -a /tmp/ekiden-single-node/mkvs_storage.leveldb.db/ /tmp/mkvs_storage.leveldb.db-imported/`

Edit: regenerate_single_node.sh script:

diff --git a/scripts/regenerate_single_node.sh b/scripts/regenerate_single_node.sh
index 4542f02f..9b41577b 100755
--- a/scripts/regenerate_single_node.sh
+++ b/scripts/regenerate_single_node.sh
@@ -39,6 +39,7 @@ ${EKIDEN_BINARY} \
     genesis init \
     --datadir ${DATADIR} \
     --debug.allow_test_keys \
+    --roothash /tmp/state-roothash.cbor \
     --debug.test_entity \
     --genesis_file ${DATADIR}/genesis_nosgx.json \
     --runtime ${DATADIR}/keymanager_genesis_nosgx.json \

https://github.com/oasislabs/ekiden/blob/master/scripts/regenerate_single_node.sh#L21

Prepare roothash.cbor:

echo "[{"header":{"version":0,"namespace":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"round":0,"timestamp":0,"header_type":0,"previous_hash":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"io_root":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"state_root":[53,23,191,128,194,46,139,37,99,25,108,134,187,96,70,41,177,127,79,207,161,72,252,118,110,128,206,230,176,78,25,7],"storage_signatures":null}}]" > /tmp/state-roothash.cbor

Regenerate config with: make regenerate-single-node

4) Prepare new node

rm -rf /tmp/ekiden-single-node; cp -R configs/single_node /tmp/ekiden-single-node && chmod -R go-rwx /tmp/ekiden-single-node 
cp -a /tmp/mkvs_storage.leveldb.db-imported/  /tmp/ekiden-single-node/mkvs_storage.leveldb.db/

5) Start node and gateway

./go/ekiden/ekiden --config configs/single_node.yml
./target/debug/gateway  --node-address "unix:/tmp/ekiden-single-node/internal.sock" --runtime-id 0000000000000000000000000000000000000000000000000000000000000000

6) Send the following transaction to crash it:

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["0xf913718203958504a817c8008313678e8080b9131c6060604052600680546001608060020a0319169055346100005760405161125c38038061125c833981016040908152815160208301519183015160608401516080850151929491929091015b600080546c01000000000000000000000000338102819004600160a060020a031992831617909255600180548884029390930492909116919091178155600380547f01000000000000000000000000000000000000000000000000000000000000008681028190046101000261ff0019828a029290920460ff19909316929092171617905561ffff831610156100e057610000565b6003805463ffff00001916620100007e010000000000000000000000000000000000000000000000000000000000008581020402179055805160028054600082905290917f405787fa12a823e0f2b7631cc41b3ba8828b3321ca811111fa75cd3aa3bb5ace602061010060018516150260001901909316849004601f90810184900482019386019083901061018057805160ff19168380011785556101ad565b828001600101855582156101ad579182015b828111156101ad578251825591602001919060010190610192565b5b506101ce9291505b808211156101ca57600081556001016101b6565b5090565b50505b50505050505b611077806101e56000396000f3606060405236156100da5760e060020a6000350463018fbf9f81146100df578063067cf182146101015780630b93381b146102335780630f79e120146102545780632038e2e61461027a57806322aab53a146102a057806323e1d8d0146102d057806328720849146102f457806332797f441461038e57806356f66f4c146103b75780638abe559314610459578063ae8421e11461047d578063b24d8de21461048c578063ba414fa6146104b3578063c6d0ffda146104d4578063e0c68db0146104f8578063ea1df43914610524578063eba37aff1461054a575b610000565b34610000576100ef600435610570565b60408051918252519081900360200190f35b3461000057610189600480803590602001908201803590602001908080602002602001604051908101604052809392919081815260200183836020028082843750506040805187358901803560208181028481018201909552818452989a99890198929750908201955093508392508501908490808284375094965061059a95505050505050565b6040518080602001806020018663ffffffff1681526020018563ffffffff168152602001846000191681526020018381038352888181518152602001915080519060200190602002808383829060006004602084601f0104600302600f01f1509050018381038252878181518152602001915080519060200190602002808383829060006004602084601f0104600302600f01f15090500197505050505050505060405180910390f35b34610000576102406109c1565b604080519115158252519081900360200190f35b34610000576102616109e4565b6040805163ffffffff9092168252519081900360200190f35b34610000576102616109f9565b6040805163ffffffff9092168252519081900360200190f35b34610000576102ad610a16565b604051808360000b81526020018260000b81526020019250505060405180910390f35b34610000576102dd610a2b565b6040805161ffff9092168252519081900360200190f35b346100005760408051602480356004818101356020818102868101820190975281865261024096833596939560449501929182919085019084908082843750506040805187358901803560208181028481018201909552818452989a998901989297509082019550935083925085019084908082843750949650610a3295505050505050565b604080519115158252519081900360200190f35b3461000057610261600435610c28565b6040805163ffffffff9092168252519081900360200190f35b3461000057610457600480803590602001908201803590602001908080602002602001604051908101604052809392919081815260200183836020028082843750506040805187358901803560208181028481018201909552818452989a99890198929750908201955093508392508501908490808284375094965050843594602081013594506040810135935060608101359250608001359050610c69565b005b34610000576102dd610f7e565b6040805161ffff9092168252519081900360200190f35b3461000057610457610f85565b005b346100005761049c600435610fc4565b6040805160009290920b8252519081900360200190f35b3461000057610240610fff565b604080519115158252519081900360200190f35b34610000576102dd611024565b6040805161ffff9092168252519081900360200190f35b346100005761026160043560243561102b565b6040805163ffffffff9092168252519081900360200190f35b3461000057610261611051565b6040805163ffffffff9092168252519081900360200190f35b346100005761026161106a565b6040805163ffffffff9092168252519081900360200190f35b600060078261ffff16815481101561000057906000526020600020900160005b505490505b919050565b60408051602081810183526000808352835180830185528181528451808401865282815285518085018752838152865194850190965282845287518751959692959394859485949293859390929091849063ffffffff8716146105fc57610000565b600354604051600096506201000090910461ffff169080591061061c5750595b8181526020808302820101604052905b5060035460405191955062010000900461ffff169080591061064b5750595b8181526020808302820101604052905b5092508560050263ffffffff166040518059106106755750595b8181526020808302820101604052905b509150600090505b8563ffffffff168161ffff16101561094d578b8161ffff168151811015610000579060200190602002015160f860020a02828260050260000161ffff16815181101561000057906020010190600160f860020a031916908160001a90535061070e8c8261ffff1681518110156100005790602001906020020151600861102b565b60f860020a02828260050260010161ffff16815181101561000057906020010190600160f860020a031916908160001a9053506107648c8261ffff1681518110156100005790602001906020020151601061102b565b60f860020a02828260050260020161ffff16815181101561000057906020010190600160f860020a031916908160001a9053506107ba8c8261ffff1681518110156100005790602001906020020151601861102b565b60f860020a02828260050260030161ffff16815181101561000057906020010190600160f860020a031916908160001a9053508c8161ffff168151811015610000579060200190602002015160f860020a02828260050260040161ffff16815181101561000057906020010190600160f860020a031916908160001a905350600360019054906101000a900460000b60000b8d8261ffff168151811015610000579060200190602002015160000b138061089b57506003548d51600091820b90910b908e9061ffff841690811015610000579060200190602002015160000b125b156109435760035460019095019461ffff620100009091041663ffffffff861611610943578b8161ffff1681518110156100005790602001906020020151846001870363ffffffff1681518110156100005763ffffffff9092166020928302909101909101528c518d9061ffff8316908110156100005790602001906020020151836001870363ffffffff1681518110156100005760009290920b6020928302909101909101525b5b5b60010161068d565b83838688600286600060405160200152604051808280519060200190808383829060006004602084601f0104600302600f01f1509050019150506020604051808303816000866161da5a03f11561000057505060405151939e50919c509a50985096505b5050505050509295509295909350565b6004546000901580156109de5750600654600063ffffffff909116115b90505b90565b600654640100000000900463ffffffff165b90565b6006546c01000000000000000000000000900463ffffffff165b90565b600354600081810b916101009004900b5b9091565b6004545b90565b60408051602081810183526000808352835191820190935282815282808080610a5b898961059a565b955095509550955095506001600780549050038a61ffff161415610bf1576004548651141580610a8e5750600554855114155b80610a9b57508551855114155b15610aa95760009650610c1a565b5060005b60045461ffff82161015610b8c5760048161ffff1681548110156100005790600052602060002090600891828204019190066004025b9054906101000a900463ffffffff1663ffffffff16868261ffff168151811015610000576020908102909101015163ffffffff1614610b255760009650610c1a565b60058161ffff1681548110156100005790600052602060002090602091828204019190065b9054906101000a900460000b60000b858261ffff168151811015610000576020908102909101015160000b14610b835760009650610c1a565b5b600101610aad565b816000191660078b61ffff16815481101561000057906000526020600020900160005b5054148015610bd0575060065463ffffffff85811664010000000090920416145b8015610be6575060065463ffffffff8481169116145b9650610c1a56610c1a565b816000191660078b61ffff16815481101561000057906000526020600020900160005b50541496505b5b5050505050509392505050565b600060048261ffff1681548110156100005790600052602060002090600891828204019190066004025b9054906101000a900463ffffffff1690505b919050565b60015460009081908190819033600160a060020a03908116911614610c8d57610000565b89518b5114610c9b57610000565b50506004548951600354919350915062010000900461ffff16825b8263ffffffff1684820363ffffffff16108015610cdc57508161ffff168163ffffffff16105b8015610cf557508863ffffffff1684820363ffffffff16105b15610e5b5760048054806001018281815481835581811511610d4c576007016008900481600701600890048360005260206000209182019101610d4c91905b80821115610d485760008155600101610d34565b5090565b5b50505091600052602060002090600891828204019190066004025b8d87850363ffffffff1681518110156100005790602001906020020151909190916101000a81548163ffffffff021916908360e060020a9081020402179055505060058054806001018281815481835581811511610dfb57601f016020900481601f01602090048360005260206000209182019101610dfb91905b80821115610d485760008155600101610d34565b5090565b5b50505091600052602060002090602091828204019190065b8c87850363ffffffff1681518110156100005790602001906020020151909190916101000a81548160ff021916908360f860020a908102040217905550505b600101610cb6565b6006805460e060020a64010000000063ffffffff8084168d01830283900463ffffffff199094169390931781810484168e018302929092040267ffffffff0000000019909116179182905568010000000000000000909104161515610ee557600680546bffffffff000000000000000019166801000000000000000060e060020a8a810204021790555b600680546fffffffff00000000000000000000000019166c0100000000000000000000000060e060020a898102040217905560078054600181018083558281838015829011610f5957600083815260209020610f599181019083015b80821115610d485760008155600101610d34565b5090565b5b505050916000526020600020900160005b50869055505b5050505050505050505050565b6007545b90565b60005433600160a060020a0390811691161480610fb0575060015433600160a060020a039081169116145b15610fc15733600160a060020a0316ff5b5b565b600060058261ffff1681548110156100005790600052602060002090602091828204019190065b9054906101000a900460000b90505b919050565b60045460009081901180156109de5750600654600063ffffffff909116115b90505b90565b6005545b90565b60008160ff1660020a63ffffffff168363ffffffff168115610000570490505b92915050565b60065468010000000000000000900463ffffffff165b90565b60065463ffffffff165b905600000000000000000000000045064e294de7b09aa180d0e1284788a21a268db8000000000000000000000000000000000000000000000000000000000000000f0000000000000000000000000000000000000000000000000000000000000019000000000000000000000000000000000000000000000000000000000000000300000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000001ba0cce47dd4b2ef2c71a5191d1453f13908456a210c1803a99466fd54069997f6aaa00ff115752da7526fae07dae4a4638fe5acb3b1566041fd2fc5fb1d656ddd2486"],"id":1}' localhost:8545
ptrus commented 5 years ago

Full backtrace (from friday):

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: HashMismatch { expected_hash: 0e154137ecc5adcb81e880c5e5a6985e3761985ba04c0b7349515f2b4116dcd1, computed_hash: e25c03c0fafff8743af08473582599de5600cb8aa32f5daaa9940c8a0035eaa5 }

stack backtrace:
   0: failure::backtrace::internal::InternalBacktrace::new::h95dfe078ee4df95c (0x559f0cff8f33)
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.5/src/backtrace/internal.rs:44
   1: failure::backtrace::Backtrace::new::h9a924cd26122cc1d (0x559f0cff8cde)
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.5/src/backtrace/mod.rs:111
   2: <failure::error::error_impl::ErrorImpl as core::convert::From<F>>::from::h84d76afa2f4348c7 (0x559f0ccfe9da)
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.5/src/error/error_impl.rs:19
   3: <failure::error::Error as core::convert::From<F>>::from::he62c92addd194dec (0x559f0cd9d623)
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.5/src/error/mod.rs:36
   4: <T as core::convert::Into<U>>::into::ha635e54711f5b1ed (0x559f0ce053c3)
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libcore/convert.rs:540
   5: <ekiden_runtime::storage::mkvs::urkel::tree::node::InternalNode as ekiden_runtime::storage::mkvs::urkel::tree::node::Node>::validate::h2ffa76c21e08fd77 (0x559f0cdac659)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/node.rs:223
   6: <ekiden_runtime::storage::mkvs::urkel::tree::node::NodeBox as ekiden_runtime::storage::mkvs::urkel::tree::node::Node>::validate::h3fdbfad47d709901 (0x559f0cdabd20)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/node.rs:79
   7: <ekiden_runtime::storage::mkvs::urkel::cache::lru_cache::LRUCache as ekiden_runtime::storage::mkvs::urkel::cache::cache::Cache>::deref_node_ptr::h05d5b6b04425ff1a (0x559f0cd29b21)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/cache/lru_cache.rs:397
   8: ekiden_runtime::storage::mkvs::urkel::tree::remove::<impl ekiden_runtime::storage::mkvs::urkel::tree::tree::UrkelTree>::_remove::h214cf6c49fa0bff0 (0x559f0cd67ca9)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/remove.rs:90
   9: ekiden_runtime::storage::mkvs::urkel::tree::remove::<impl ekiden_runtime::storage::mkvs::urkel::tree::tree::UrkelTree>::_remove::h214cf6c49fa0bff0 (0x559f0cd672af)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/remove.rs:71
  10: ekiden_runtime::storage::mkvs::urkel::tree::remove::<impl ekiden_runtime::storage::mkvs::urkel::tree::tree::UrkelTree>::_remove::h214cf6c49fa0bff0 (0x559f0cd672af)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/remove.rs:71
  11: ekiden_runtime::storage::mkvs::urkel::tree::remove::<impl ekiden_runtime::storage::mkvs::urkel::tree::tree::UrkelTree>::_remove::h214cf6c49fa0bff0 (0x559f0cd672af)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/remove.rs:71
  12: ekiden_runtime::storage::mkvs::urkel::tree::remove::<impl ekiden_runtime::storage::mkvs::urkel::tree::tree::UrkelTree>::_remove::h214cf6c49fa0bff0 (0x559f0cd672af)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/remove.rs:71
  13: ekiden_runtime::storage::mkvs::urkel::tree::remove::<impl ekiden_runtime::storage::mkvs::urkel::tree::tree::UrkelTree>::_remove::h214cf6c49fa0bff0 (0x559f0cd672af)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/remove.rs:71
  14: ekiden_runtime::storage::mkvs::urkel::tree::remove::<impl ekiden_runtime::storage::mkvs::urkel::tree::tree::UrkelTree>::_remove::h214cf6c49fa0bff0 (0x559f0cd672af)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/remove.rs:71
  15: ekiden_runtime::storage::mkvs::urkel::tree::remove::<impl ekiden_runtime::storage::mkvs::urkel::tree::tree::UrkelTree>::_remove::h214cf6c49fa0bff0 (0x559f0cd672af)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/remove.rs:71
  16: ekiden_runtime::storage::mkvs::urkel::tree::remove::<impl ekiden_runtime::storage::mkvs::urkel::tree::tree::UrkelTree>::_remove::h214cf6c49fa0bff0 (0x559f0cd672af)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/remove.rs:71
  17: ekiden_runtime::storage::mkvs::urkel::tree::remove::<impl ekiden_runtime::storage::mkvs::urkel::tree::tree::UrkelTree>::remove::hab7b20129276f22f (0x559f0cd66298)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/remove.rs:18
  18: ekiden_runtime::storage::mkvs::urkel::tree::mkvs::<impl ekiden_runtime::storage::mkvs::MKVS for ekiden_runtime::storage::mkvs::urkel::tree::tree::UrkelTree>::remove::hb8581721bb434ff0 (0x559f0cd65d4b)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/mkvs.rs:29
  19: <runtime_ethereum_common::storage::ThreadLocalMKVS as ethcore::mkvs::MKVS>::remove::{{closure}}::h9c40800d050a4bc9 (0x559f0c1ea0db)
             at common/src/storage.rs:40
  20: ekiden_runtime::storage::context::StorageContext::with_current::{{closure}}::h80dcfdb33ba8dbaa (0x559f0c14175b)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/context.rs:73
  21: std::thread::local::LocalKey<T>::try_with::hdb6ea5a3fa209a0a (0x559f0c2011fe)
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libstd/thread/local.rs:299
  22: std::thread::local::LocalKey<T>::with::h0f482d07cb43a1a6 (0x559f0c1ff76c)
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libstd/thread/local.rs:245
  23: ekiden_runtime::storage::context::StorageContext::with_current::he4037250279c4b1e (0x559f0c1415ae)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/context.rs:68
  24: <runtime_ethereum_common::storage::ThreadLocalMKVS as ethcore::mkvs::MKVS>::remove::h522b17245e5029f1 (0x559f0c12671f)
             at common/src/storage.rs:39
  25: <alloc::boxed::Box<T> as ethcore::mkvs::MKVS>::remove::haa3d43bc5b97d80b (0x559f0c292741)
             at /root/.cargo/git/checkouts/parity-37ea6480f4a851e5/6195938/ethcore/src/mkvs.rs:38
  26: <ethcore::mkvs::PrefixedMKVS as ethcore::mkvs::MKVS>::remove::hdd41318d2758acb7 (0x559f0c25b211)
             at /root/.cargo/git/checkouts/parity-37ea6480f4a851e5/6195938/ethcore/src/mkvs.rs:91
  27: ethcore::state::account::Account::commit_storage::h86fe4a2a8f4a909f (0x559f0c2bff7d)
             at /root/.cargo/git/checkouts/parity-37ea6480f4a851e5/6195938/ethcore/src/state/account.rs:456
  28: ethcore::state::State<B>::commit::hdf8083ab0878c105 (0x559f0c0e492a)
             at /root/.cargo/git/checkouts/parity-37ea6480f4a851e5/6195938/ethcore/src/state/mod.rs:985
  29: <runtime_ethereum::block::EthereumBatchHandler as ekiden_runtime::transaction::dispatcher::BatchHandler>::end_batch::hdbe69ca68704a981 (0x559f0c0ccf10)
             at src/block.rs:84
  30: ekiden_runtime::transaction::dispatcher::Dispatcher::dispatch_batch::h902143a42ac18b74 (0x559f0cdaf03d)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/transaction/dispatcher.rs:255
  31: ekiden_runtime::dispatcher::Dispatcher::dispatch_txn::{{closure}}::h8de6abe99ee46b75 (0x559f0ce72f9f)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/dispatcher.rs:271
  32: ekiden_runtime::storage::context::StorageContext::enter::hb2721b0ba2e72022 (0x559f0cdab56c)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/context.rs:56
  33: ekiden_runtime::dispatcher::Dispatcher::dispatch_txn::hfcfbb5dcc87a5f4a (0x559f0ce71534)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/dispatcher.rs:270
  34: ekiden_runtime::dispatcher::Dispatcher::run::h59bb5f05a175059f (0x559f0ce6ffbd)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/dispatcher.rs:203
  35: ekiden_runtime::dispatcher::Dispatcher::new::{{closure}}::h1b5c699548a97330 (0x559f0ce6ee94)
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/dispatcher.rs:103
  36: std::sys_common::backtrace::__rust_begin_short_backtrace::h93a66bfaa8de654d (0x559f0ce011f2)
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libstd/sys_common/backtrace.rs:77
  37: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::h9b687b100072f157 (0x559f0cca8641)
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libstd/thread/mod.rs:470
  38: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h83e1e0c858bdc77c (0x559f0cdf5871)
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libstd/panic.rs:309
  39: std::panicking::try::do_call::h1bb2d27ca4cfa8ae (0x559f0cd92f07)
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libstd/panicking.rs:293
  40: __rust_maybe_catch_panic (0x559f0d0f2dca)
             at src/libpanic_unwind/lib.rs:85
  41: std::panicking::try::h21551824ef370b8e (0x559f0cd92d50)
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libstd/panicking.rs:272
  42: std::panic::catch_unwind::h92cd95e8b9a1c2dc (0x559f0ce030e3)
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libstd/panic.rs:388
  43: std::thread::Builder::spawn_unchecked::{{closure}}::h1d46970a95911467 (0x559f0cca8432)
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libstd/thread/mod.rs:469
  44: core::ops::function::FnOnce::call_once{{vtable.shim}}::h3c31adb2e88e2744 (0x559f0cd3b394)
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libcore/ops/function.rs:231
  45: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h08bd7d4edd708d24 (0x559f0d0e72cf)
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/liballoc/boxed.rs:746
  46: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::he2a07dcda49e9385 (0x559f0d0f2530)
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/liballoc/boxed.rs:746
      std::sys_common::thread::start_thread::h9f229612a1fc41e1
             at src/libstd/sys_common/thread.rs:13
      std::sys::unix::thread::Thread::new::thread_start::h5b2c91946063a815
             at src/libstd/sys/unix/thread.rs:79
  47: start_thread (0x7f428d37e6db)
  48: __clone (0x7f428ce8f88f)
  49: <unknown> (0x0)', src/libcore/result.rs:999:5
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.25/src/backtrace/libunwind.rs:97
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.25/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:47
   3: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:36
   4: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:197
   5: std::panicking::default_hook
             at src/libstd/panicking.rs:211
   6: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:474
   7: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:381
   8: rust_begin_unwind
             at src/libstd/panicking.rs:308
   9: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
  10: core::result::unwrap_failed
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libcore/macros.rs:18
  11: core::result::Result<T,E>::unwrap
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libcore/result.rs:800
  12: ekiden_runtime::storage::mkvs::urkel::tree::mkvs::<impl ekiden_runtime::storage::mkvs::MKVS for ekiden_runtime::storage::mkvs::urkel::tree::tree::UrkelTree>::remove
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/mkvs/urkel/tree/mkvs.rs:29
  13: <runtime_ethereum_common::storage::ThreadLocalMKVS as ethcore::mkvs::MKVS>::remove::{{closure}}
             at common/src/storage.rs:40
  14: ekiden_runtime::storage::context::StorageContext::with_current::{{closure}}
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/context.rs:73
  15: std::thread::local::LocalKey<T>::try_with
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libstd/thread/local.rs:299
  16: std::thread::local::LocalKey<T>::with
             at /rustc/37d001e4deb206ed954fde5d91690221e8306fc3/src/libstd/thread/local.rs:245
  17: ekiden_runtime::storage::context::StorageContext::with_current
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/context.rs:68
  18: <runtime_ethereum_common::storage::ThreadLocalMKVS as ethcore::mkvs::MKVS>::remove
             at common/src/storage.rs:39
  19: <alloc::boxed::Box<T> as ethcore::mkvs::MKVS>::remove
             at /root/.cargo/git/checkouts/parity-37ea6480f4a851e5/6195938/ethcore/src/mkvs.rs:38
  20: <ethcore::mkvs::PrefixedMKVS as ethcore::mkvs::MKVS>::remove
             at /root/.cargo/git/checkouts/parity-37ea6480f4a851e5/6195938/ethcore/src/mkvs.rs:91
  21: ethcore::state::account::Account::commit_storage
             at /root/.cargo/git/checkouts/parity-37ea6480f4a851e5/6195938/ethcore/src/state/account.rs:456
  22: ethcore::state::State<B>::commit
             at /root/.cargo/git/checkouts/parity-37ea6480f4a851e5/6195938/ethcore/src/state/mod.rs:985
  23: <runtime_ethereum::block::EthereumBatchHandler as ekiden_runtime::transaction::dispatcher::BatchHandler>::end_batch
             at src/block.rs:84
  24: ekiden_runtime::transaction::dispatcher::Dispatcher::dispatch_batch
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/transaction/dispatcher.rs:255
  25: ekiden_runtime::dispatcher::Dispatcher::dispatch_txn::{{closure}}
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/dispatcher.rs:271
  26: ekiden_runtime::storage::context::StorageContext::enter
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/storage/context.rs:56
  27: ekiden_runtime::dispatcher::Dispatcher::dispatch_txn
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/dispatcher.rs:270
  28: ekiden_runtime::dispatcher::Dispatcher::run
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/dispatcher.rs:203
  29: ekiden_runtime::dispatcher::Dispatcher::new::{{closure}}
             at /root/.cargo/git/checkouts/ekiden-95b12fc3add89d55/3c78f1b/runtime/src/dispatcher.rs:103
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
kostko commented 5 years ago

Should be fixed by #1886.

ptrus commented 5 years ago

Confirmed https://github.com/oasislabs/ekiden/pull/1886 fixed this