search

stacks-network / stacks-core

The Stacks blockchain implementation
https://docs.stacks.co
GNU General Public License v3.0
3.01k stars 671 forks source link

Deadlock on restart with working_dir #2546

Closed zone117x closed 2 years ago

zone117x commented 3 years ago

Around half the time when restarting everything works as expected. Half the time the node deadlocks.

lldb dump:

$ lldb --attach-pid 51168
(lldb) process attach --pid 51168
Process 51168 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff68d6d756 libsystem_kernel.dylib`__semwait_signal + 10
libsystem_kernel.dylib`__semwait_signal:
->  0x7fff68d6d756 <+10>: jae    0x7fff68d6d760            ; <+20>
    0x7fff68d6d758 <+12>: movq   %rax, %rdi
    0x7fff68d6d75b <+15>: jmp    0x7fff68d6c22d            ; cerror
    0x7fff68d6d760 <+20>: retq   
Target 0: (stacks-node) stopped.

Executable module set to "/stacks-node".
Architecture set to: x86_64h-apple-macosx-.
(lldb) bt
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff68d6d756 libsystem_kernel.dylib`__semwait_signal + 10
    frame #1: 0x00007fff68cf0eea libsystem_c.dylib`nanosleep + 196
    frame #2: 0x0000000109736600 stacks-node`std::thread::sleep::h73abd6b6c0d95d1d [inlined] std::sys::unix::thread::Thread::sleep::h92abcea5a6a22108 at thread.rs:164:20 [opt]
    frame #3: 0x00000001097365c0 stacks-node`std::thread::sleep::h73abd6b6c0d95d1d at mod.rs:792 [opt]
    frame #4: 0x000000010807bfeb stacks-node`blockstack_lib::util::sleep_ms::h194300d1a297d785(millis=100) at mod.rs:56:5
    frame #5: 0x0000000107cc7a01 stacks-node`stacks_node::burnchains::bitcoin_regtest_controller::BitcoinRegtestController::wait_for_sortitions::hbc014834b27c4857(self=0x00007ffee7fb43a0, height_to_wait=Option<u64> @ 0x00007ffee7fb25a0) at bitcoin_regtest_controller.rs:1324:13
    frame #6: 0x0000000107cf0862 stacks-node`stacks_node::run_loop::neon::RunLoop::start::h32fa40097d573f88(self=0x00007ffee7fc2a90, burnchain_opt=Option<blockstack_lib::burnchains::Burnchain> @ 0x00007ffee7fc33c0, mine_start=0) at neon.rs:272:33
    frame #7: 0x0000000107e208c7 stacks-node`stacks_node::main::h437665d13b6cd0a7 at main.rs:158:9
    frame #8: 0x0000000107daeaee stacks-node`core::ops::function::FnOnce::call_once::h5221dc738099f80d((null)=(stacks-node`stacks_node::main::h437665d13b6cd0a7 at main.rs:54), (null)=<unavailable>) at function.rs:227:5
    frame #9: 0x0000000107dc2c71 stacks-node`std::sys_common::backtrace::__rust_begin_short_backtrace::h2a25c0ef8f2b7d22(f=(stacks-node`stacks_node::main::h437665d13b6cd0a7 at main.rs:54)) at backtrace.rs:125:18
    frame #10: 0x0000000107e30d84 stacks-node`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h3f3f9c822ccc5691 at rt.rs:66:18
    frame #11: 0x00000001097468c4 stacks-node`std::rt::lang_start_internal::h36ccce6e8a047133 [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::h81f763a559f81b5c at function.rs:259:13 [opt]
    frame #12: 0x00000001097468bd stacks-node`std::rt::lang_start_internal::h36ccce6e8a047133 [inlined] std::panicking::try::do_call::h8f7e7501b4b6c841 at panicking.rs:381 [opt]
    frame #13: 0x00000001097468bd stacks-node`std::rt::lang_start_internal::h36ccce6e8a047133 [inlined] std::panicking::try::h5def58989e0f2c84 at panicking.rs:345 [opt]
    frame #14: 0x00000001097468bd stacks-node`std::rt::lang_start_internal::h36ccce6e8a047133 [inlined] std::panic::catch_unwind::h8210e1d7d92e8f91 at panic.rs:396 [opt]
    frame #15: 0x00000001097468bd stacks-node`std::rt::lang_start_internal::h36ccce6e8a047133 at rt.rs:51 [opt]
    frame #16: 0x0000000107e30d61 stacks-node`std::rt::lang_start::h432d49275fdf4aac(main=(stacks-node`stacks_node::main::h437665d13b6cd0a7 at main.rs:54), argc=3, argv=0x00007ffee7fc3700) at rt.rs:65:5
    frame #17: 0x0000000107e20d92 stacks-node`main + 34
    frame #18: 0x00007fff68c29cc9 libdyld.dylib`start + 1
(lldb)

Debug log:

$ STACKS_EVENT_OBSERVER=localhost:3700 BLOCKSTACK_DEBUG=1 ./stacks-node start --config=/Users/matt/Projects/stacks-blockchain-api/stacks-blockchain/Stacks-follower.toml
INFO [1616665965.520358] [testnet/stacks-node/src/main.rs:80] [main] stacks-node 0.1.0 (develop:18243c1d7, release build, macos [x86_64])
INFO [1616665965.520410] [testnet/stacks-node/src/main.rs:125] [main] Loading config at path /Users/matt/Projects/stacks-blockchain-api/stacks-blockchain/Stacks-follower.toml
DEBG [1616665965.528751] [testnet/stacks-node/src/main.rs:139] [main] node configuration NodeConfig { name: "helium-node", seed: [189, 155, 201, 8, 114, 235, 31, 116, 131, 93, 71, 247, 18, 160, 46, 209, 77, 190, 32, 15, 158, 164, 216, 136, 32, 133, 247, 135, 57, 26, 66, 58], working_dir: "/tmp/stacks-blockchain-data", rpc_bind: "0.0.0.0:20443", p2p_bind: "0.0.0.0:20444", data_url: "http://0.0.0.0:20443", p2p_address: "0.0.0.0:20443", local_peer_seed: [206, 135, 199, 24, 244, 138, 10, 214, 40, 31, 40, 84, 151, 215, 59, 186, 188, 212, 166, 127, 188, 203, 75, 78, 7, 178, 198, 130, 201, 86, 209, 106], bootstrap_node: [Neighbor { addr: 18000000+00000001://35.199.6.155:20444, public_key: Secp256k1PublicKey { key: PublicKey(6bd3594422e9c3d18da9bef6fef3f21034b978076743a33783ae70c74a467ada16e2af79bbc8d4b9e59b6614b34e6243ddb863f1cc75fedf99ab74453781959d), compressed: true }, expire_block: 9999999, last_contact_time: 0, allowed: 0, denied: 0, asn: 0, org: 0, in_degree: 0, out_degree: 0 }, Neighbor { addr: 18000000+00000001://34.86.228.189:20444, public_key: Secp256k1PublicKey { key: PublicKey(1635b6447343cfd908648348dd52e041a6b4bc5ff7dd009ac9f8b5aa22e5eaafee0940fc8dc53c47858e392846f782ce5e9facfd68f751492f9c4a51a722adca), compressed: true }, expire_block: 9999999, last_contact_time: 0, allowed: 0, denied: 0, asn: 0, org: 0, in_degree: 0, out_degree: 0 }, Neighbor { addr: 18000000+00000001://35.245.226.0:20444, public_key: Secp256k1PublicKey { key: PublicKey(624e4ab34063c3637d9f99b9a72930991978e5065ca283cdd40ebe76ea1222653303f61e24a5cf2dc2ab782043a10d26c3b64e4b2517b954a0e92ee2e926dc25), compressed: true }, expire_block: 9999999, last_contact_time: 0, allowed: 0, denied: 0, asn: 0, org: 0, in_degree: 0, out_degree: 0 }], deny_nodes: [], miner: false, mock_mining: false, mine_microblocks: true, microblock_frequency: 30000, max_microblocks: 65535, wait_time_for_microblocks: 30000, prometheus_bind: None, pox_sync_sample_secs: 30, use_test_genesis_chainstate: None }
DEBG [1616665965.528876] [testnet/stacks-node/src/main.rs:140] [main] burnchain configuration BurnchainConfig { chain: "bitcoin", mode: "mainnet", chain_id: 1, peer_version: 402653184, commit_anchor_block_within: 5000, burn_fee_cap: 20000, peer_host: "34.86.174.130", peer_port: 8333, rpc_port: 8332, rpc_ssl: false, username: Some("blockstack"), password: Some("blockstacksystem"), timeout: 300, magic_bytes: MagicBytes([88, 50]), local_mining_public_key: None, process_exit_at_block_height: None, poll_time_secs: 10, satoshis_per_byte: 50, leader_key_tx_estimated_size: 290, block_commit_tx_estimated_size: 350, rbf_fee_increment: 5 }
DEBG [1616665965.528911] [testnet/stacks-node/src/main.rs:141] [main] connection configuration ConnectionOptions { inbox_maxlen: 100, outbox_maxlen: 100, connect_timeout: 10, handshake_timeout: 30, timeout: 15, idle_timeout: 15, heartbeat: 3600, private_key_lifetime: 9223372036854775807, num_neighbors: 16, num_clients: 750, soft_num_neighbors: 16, soft_num_clients: 750, max_neighbors_per_host: 1, max_clients_per_host: 4, soft_max_neighbors_per_host: 1, soft_max_neighbors_per_org: 32, soft_max_clients_per_host: 4, max_neighbors_of_neighbor: 10, max_http_clients: 1000, neighbor_request_timeout: 30, num_initial_walks: 10, walk_retry_count: 10, walk_interval: 60, walk_inbound_ratio: 2, walk_min_duration: 20, walk_max_duration: 40, walk_reset_prob: 0.05, walk_reset_interval: 600, walk_state_timeout: 60, inv_sync_interval: 45, download_interval: 10, pingback_timeout: 60, dns_timeout: 15000, max_inflight_blocks: 6, max_inflight_attachments: 6, max_attachment_retry_count: 32, read_only_call_limit: ExecutionCost { write_length: 0, write_count: 0, read_length: 100000, read_count: 30, runtime: 1000000000 }, maximum_call_argument_size: 83886080, max_block_push_bandwidth: 0, max_microblocks_push_bandwidth: 0, max_transaction_push_bandwidth: 0, max_sockets: 800, public_ip_address: None, public_ip_request_timeout: 60, public_ip_timeout: 3600, public_ip_max_retries: 3, max_block_push: 10, max_microblock_push: 10, antientropy_retry: 86400, max_buffered_blocks_available: 1, max_buffered_microblocks_available: 1, max_buffered_blocks: 1, max_buffered_microblocks: 10, disable_neighbor_walk: false, disable_chat_neighbors: false, disable_inv_sync: false, disable_inv_chat: false, disable_block_download: false, disable_network_prune: false, disable_network_bans: false, disable_block_advertisement: false, disable_pingbacks: false, disable_inbound_walks: false, disable_natpunch: false, disable_inbound_handshakes: false, force_disconnect_interval: None }
DEBG [1616665965.529142] [testnet/stacks-node/src/main.rs:142] [main] block_limit ExecutionCost { write_length: 15000000, write_count: 7750, read_length: 100000000, read_count: 7750, runtime: 5000000000 }
DEBG [1616665965.531445] [src/burnchains/bitcoin/spv.rs:479] [main] Initialized block headers at /tmp/stacks-blockchain-data/mainnet/headers.sqlite
INFO [1616665965.531734] [testnet/stacks-node/src/run_loop/neon.rs:153] [main] Will run as a Follower node
INFO [1616665965.531754] [testnet/stacks-node/src/run_loop/neon.rs:160] [main] Start syncing Bitcoin headers, feel free to grab a cup of coffee, this can take a while
DEBG [1616665965.532862] [src/chainstate/burn/db/sortdb.rs:2134] [main] Connect/Open  sortdb '/tmp/stacks-blockchain-data/mainnet/burnchain/sortition/data.sqlite' as 'readwrite', with index as '/tmp/stacks-blockchain-data/mainnet/burnchain/sortition/marf.sqlite'
DEBG [1616665965.533000] [src/chainstate/stacks/index/storage.rs:815] [main] Opened TrieFileStorage /tmp/stacks-blockchain-data/mainnet/burnchain/sortition/marf.sqlite;
DEBG [1616665965.542197] [src/burnchains/bitcoin/spv.rs:479] [main] Initialized block headers at /tmp/stacks-blockchain-data/mainnet/headers.sqlite.reorg
DEBG [1616665965.542318] [src/burnchains/bitcoin/spv.rs:648] [main] Insert 1 headers to /tmp/stacks-blockchain-data/mainnet/headers.sqlite.reorg in range 676207-676208
DEBG [1616665965.542387] [src/burnchains/bitcoin/spv.rs:674] [main] No header for child block 676208, so will not validate continuity
DEBG [1616665965.542429] [src/burnchains/bitcoin/spv.rs:691] [main] No header for parent block 676206, so will not validate continuity
DEBG [1616665965.542435] [src/burnchains/bitcoin/spv.rs:565] [main] Write 1 headers at /tmp/stacks-blockchain-data/mainnet/headers.sqlite.reorg at 676208
DEBG [1616665965.543245] [src/burnchains/bitcoin/indexer.rs:604] [main] Search for reorg'ed Bitcoin headers from 676208 - 676224
DEBG [1616665965.543261] [src/burnchains/bitcoin/indexer.rs:428] [main] (Re)establish peer connection
DEBG [1616665965.647776] [src/burnchains/bitcoin/network.rs:156] [main] Begin peer handshake to 34.86.174.130:8333
DEBG [1616665965.647830] [src/burnchains/bitcoin/network.rs:256] [main] Send version (nonce=8147568093942623740) to 34.86.174.130:8333
DEBG [1616665965.839877] [src/burnchains/bitcoin/network.rs:267] [main] Handle version -- remote peer blockchain height is 676223
DEBG [1616665965.839894] [src/burnchains/bitcoin/network.rs:285] [main] Send verack
DEBG [1616665965.840041] [src/burnchains/bitcoin/network.rs:294] [main] Handle verack
DEBG [1616665965.840052] [src/burnchains/bitcoin/network.rs:167] [main] Established connection to 34.86.174.130:8333, who has 676223 blocks
DEBG [1616665965.840062] [src/burnchains/bitcoin/spv.rs:861] [main] Get headers 676208-676223 to /tmp/stacks-blockchain-data/mainnet/headers.sqlite.reorg
DEBG [1616665965.840187] [src/burnchains/bitcoin/network.rs:344] [main] Send GetHeaders 000000000000000000005270648cc73d1cb8170f300568f8082b5fc356f875bc for 2000 headers to 34.86.174.130:8333
DEBG [1616665965.946076] [src/burnchains/bitcoin/network.rs:308] [main] Handle ping 9464019437434232736
DEBG [1616665965.946097] [src/burnchains/bitcoin/network.rs:311] [main] Send pong 9464019437434232736
DEBG [1616665965.946281] [src/burnchains/bitcoin/indexer.rs:483] [main] Unhandled message Addr([(1616665965, Address {services: 1037, address: [0, 0, 0, 0, 0, 65535, 8790, 44674], port: 8333})])
DEBG [1616665965.949988] [src/burnchains/bitcoin/spv.rs:648] [main] Insert 15 headers to /tmp/stacks-blockchain-data/mainnet/headers.sqlite.reorg in range 676208-676223
DEBG [1616665965.950177] [src/burnchains/bitcoin/spv.rs:674] [main] No header for child block 676223, so will not validate continuity
DEBG [1616665965.950247] [src/burnchains/bitcoin/spv.rs:565] [main] Write 15 headers at /tmp/stacks-blockchain-data/mainnet/headers.sqlite.reorg at 676209
DEBG [1616665965.951670] [src/burnchains/bitcoin/spv.rs:291] [main] Validate PoW between blocks 677376-679392
DEBG [1616665965.951763] [src/burnchains/bitcoin/spv.rs:546] [main] Handled 15 Headers: 00000000000000000006ba9bb7e3ea23f05a69602a80de7aadbe9fdbcf76074d-00000000000000000009617805024b72e826308cc41c80a032c5d07170df16fe
DEBG [1616665965.951832] [src/burnchains/bitcoin/spv.rs:928] [main] Request headers for blocks 676223 - 678223 in range 676208 - 676223
DEBG [1616665965.951897] [src/burnchains/bitcoin/network.rs:344] [main] Send GetHeaders 00000000000000000009617805024b72e826308cc41c80a032c5d07170df16fe for 2000 headers to 34.86.174.130:8333
DEBG [1616665966.060025] [src/burnchains/bitcoin/indexer.rs:473] [main] Message handler indicates to stop
DEBG [1616665966.060661] [src/burnchains/bitcoin/indexer.rs:713] [main] Found common Bitcoin block ancestor at height 676223: BlockHeader { version: 671080448, prev_blockhash: 33580c487cb50b1f18dc65b9312c93094166606c24a305000000000000000000, merkle_root: 3cc5f94fe478826c1d9514c2318cc30ba377eb297fba0dac9698b5ed1c7599e8, time: 1616665381, bits: 386719599, nonce: 1650155830 }
DEBG [1616665966.060715] [src/burnchains/bitcoin/indexer.rs:749] [main] Bitcoin headers history is consistent up to 676223
DEBG [1616665966.061132] [src/burnchains/burnchain.rs:1216] [main] Sync headers from 676223
DEBG [1616665966.061154] [src/burnchains/bitcoin/indexer.rs:517] [main] Sync all headers starting at block 676223
DEBG [1616665966.063183] [src/burnchains/bitcoin/spv.rs:479] [main] Initialized block headers at /tmp/stacks-blockchain-data/mainnet/headers.sqlite
DEBG [1616665966.063209] [src/burnchains/bitcoin/indexer.rs:428] [main] (Re)establish peer connection
DEBG [1616665966.167099] [src/burnchains/bitcoin/network.rs:156] [main] Begin peer handshake to 34.86.174.130:8333
DEBG [1616665966.167135] [src/burnchains/bitcoin/network.rs:256] [main] Send version (nonce=8147568093942623740) to 34.86.174.130:8333
DEBG [1616665966.377648] [src/burnchains/bitcoin/network.rs:267] [main] Handle version -- remote peer blockchain height is 676223
DEBG [1616665966.377686] [src/burnchains/bitcoin/network.rs:285] [main] Send verack
DEBG [1616665966.377912] [src/burnchains/bitcoin/network.rs:294] [main] Handle verack
DEBG [1616665966.377925] [src/burnchains/bitcoin/network.rs:167] [main] Established connection to 34.86.174.130:8333, who has 676223 blocks
DEBG [1616665966.377939] [src/burnchains/bitcoin/spv.rs:857] [main] Have all headers up to 676223
DEBG [1616665966.377950] [src/burnchains/bitcoin/indexer.rs:449] [main] begin_session() terminates conversation
DEBG [1616665966.378172] [src/burnchains/burnchain.rs:1225] [main] Sync'ed headers from 666212 to 676223. DB at 666212
DEBG [1616665966.378188] [src/burnchains/burnchain.rs:1232] [main] Will download up to max burn block height 666050
DEBG [1616665966.378884] [src/burnchains/burnchain.rs:1255] [main] Nothing to do; already have blocks up to 666050
DEBG [1616665966.382582] [src/chainstate/burn/db/sortdb.rs:2089] [main] Open sortdb '/tmp/stacks-blockchain-data/mainnet/burnchain/sortition/data.sqlite' as 'readwrite', with index as '/tmp/stacks-blockchain-data/mainnet/burnchain/sortition/marf.sqlite'
DEBG [1616665966.382764] [src/chainstate/stacks/index/storage.rs:815] [main] Opened TrieFileStorage /tmp/stacks-blockchain-data/mainnet/burnchain/sortition/marf.sqlite;
DEBG [1616665966.385197] [testnet/stacks-node/src/burnchains/bitcoin_regtest_controller.rs:462] [main] Done receiving blocks
INFO [1616665966.385292] [testnet/stacks-node/src/event_dispatcher.rs:632] [main] Registering event observer at: localhost:3700
DEBG [1616665966.391140] [src/chainstate/stacks/index/storage.rs:815] [main] Opened TrieFileStorage /tmp/stacks-blockchain-data/mainnet/chainstate/vm/index.sqlite;
DEBG [1616665966.392712] [src/chainstate/stacks/index/storage.rs:815] [main] Opened TrieFileStorage /tmp/stacks-blockchain-data/mainnet/chainstate/vm/clarity/marf.sqlite;
DEBG [1616665966.395420] [src/chainstate/burn/db/sortdb.rs:2089] [chains-coordinator] Open sortdb '/tmp/stacks-blockchain-data/mainnet/burnchain/sortition/data.sqlite' as 'readwrite', with index as '/tmp/stacks-blockchain-data/mainnet/burnchain/sortition/marf.sqlite'
DEBG [1616665966.395753] [src/chainstate/stacks/index/storage.rs:815] [chains-coordinator] Opened TrieFileStorage /tmp/stacks-blockchain-data/mainnet/burnchain/sortition/marf.sqlite;
lgalabru commented 3 years ago

If you can reproduce, can you please provide the stack trace for all the threads? bt all Thanks!

zone117x commented 3 years ago

All threads:

$ lldb --attach-pid 22603
(lldb) process attach --pid 22603
Process 22603 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff68d6d756 libsystem_kernel.dylib`__semwait_signal + 10
libsystem_kernel.dylib`__semwait_signal:
->  0x7fff68d6d756 <+10>: jae    0x7fff68d6d760            ; <+20>
    0x7fff68d6d758 <+12>: movq   %rax, %rdi
    0x7fff68d6d75b <+15>: jmp    0x7fff68d6c22d            ; cerror
    0x7fff68d6d760 <+20>: retq   
Target 0: (stacks-node) stopped.

Executable module set to "/Users/matt/Projects/stacks-blockchain-merging/target/release/stacks-node".
Architecture set to: x86_64h-apple-macosx-.
(lldb) bt all
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff68d6d756 libsystem_kernel.dylib`__semwait_signal + 10
    frame #1: 0x00007fff68cf0eea libsystem_c.dylib`nanosleep + 196
    frame #2: 0x00000001098df7b0 stacks-node`std::thread::sleep::h73abd6b6c0d95d1d [inlined] std::sys::unix::thread::Thread::sleep::h92abcea5a6a22108 at thread.rs:164:20 [opt]
    frame #3: 0x00000001098df770 stacks-node`std::thread::sleep::h73abd6b6c0d95d1d at mod.rs:792 [opt]
    frame #4: 0x0000000108d1a627 stacks-node`stacks_node::burnchains::bitcoin_regtest_controller::BitcoinRegtestController::wait_for_sortitions::hbc014834b27c4857(self=0x00007ffee703f878, height_to_wait=Option<u64> @ 0x00007facac0eb480) at bitcoin_regtest_controller.rs:1327:13 [opt]
    frame #5: 0x0000000108c3b66f stacks-node`stacks_node::run_loop::neon::RunLoop::start::hcd08cb6086a1c1e9(self=<unavailable>, burnchain_opt=Option<blockstack_lib::burnchains::Burnchain> @ 0x00007ffee7047030, mine_start=<unavailable>) at neon.rs:272:33 [opt]
    frame #6: 0x0000000108cae674 stacks-node`stacks_node::main::h437665d13b6cd0a7 at main.rs:158:9 [opt]
    frame #7: 0x0000000108d5c57a stacks-node`std::sys_common::backtrace::__rust_begin_short_backtrace::h2a25c0ef8f2b7d22 [inlined] core::ops::function::FnOnce::call_once::h5221dc738099f80d((null)=<unavailable>) at function.rs:227:5 [opt]
    frame #8: 0x0000000108d5c578 stacks-node`std::sys_common::backtrace::__rust_begin_short_backtrace::h2a25c0ef8f2b7d22(f=<unavailable>) at backtrace.rs:125 [opt]
    frame #9: 0x0000000108d6240c stacks-node`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h3f3f9c822ccc5691 at rt.rs:66:18 [opt]
    frame #10: 0x00000001098ef9d4 stacks-node`std::rt::lang_start_internal::h36ccce6e8a047133 [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::h81f763a559f81b5c at function.rs:259:13 [opt]
    frame #11: 0x00000001098ef9cd stacks-node`std::rt::lang_start_internal::h36ccce6e8a047133 [inlined] std::panicking::try::do_call::h8f7e7501b4b6c841 at panicking.rs:381 [opt]
    frame #12: 0x00000001098ef9cd stacks-node`std::rt::lang_start_internal::h36ccce6e8a047133 [inlined] std::panicking::try::h5def58989e0f2c84 at panicking.rs:345 [opt]
    frame #13: 0x00000001098ef9cd stacks-node`std::rt::lang_start_internal::h36ccce6e8a047133 [inlined] std::panic::catch_unwind::h8210e1d7d92e8f91 at panic.rs:396 [opt]
    frame #14: 0x00000001098ef9cd stacks-node`std::rt::lang_start_internal::h36ccce6e8a047133 at rt.rs:51 [opt]
    frame #15: 0x0000000108caf1c9 stacks-node`main + 41
    frame #16: 0x00007fff68c29cc9 libdyld.dylib`start + 1
  thread #2, name = 'ctrl-c'
    frame #0: 0x00007fff68d6b81e libsystem_kernel.dylib`read + 10
    frame #1: 0x0000000108dc13fb stacks-node`nix::unistd::read::h0ac1dc01f7a47096(fd=<unavailable>, buf=<unavailable>) at unistd.rs:991:24 [opt]
    frame #2: 0x0000000108d75921 stacks-node`ctrlc::set_handler::_$u7b$$u7b$closure$u7d$$u7d$::h813672e3b192b3b8 at mod.rs:139:15 [opt]
    frame #3: 0x0000000108d75909 stacks-node`ctrlc::set_handler::_$u7b$$u7b$closure$u7d$$u7d$::h813672e3b192b3b8 at lib.rs:111 [opt]
    frame #4: 0x0000000108d5f7e9 stacks-node`std::sys_common::backtrace::__rust_begin_short_backtrace::hddcde870272be07d(f=<unavailable>) at backtrace.rs:125:18 [opt]
    frame #5: 0x0000000108d664c4 stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h50d48fa4ff8a97dd [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hb1000583e7396e8b at mod.rs:474:17 [opt]
    frame #6: 0x0000000108d664bf stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h50d48fa4ff8a97dd [inlined] _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h1cb490d06fd1ce23(self=<unavailable>) at panic.rs:322 [opt]
    frame #7: 0x0000000108d664bf stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h50d48fa4ff8a97dd at panicking.rs:381 [opt]
    frame #8: 0x0000000108d664bf stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h50d48fa4ff8a97dd at panicking.rs:345 [opt]
    frame #9: 0x0000000108d664bf stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h50d48fa4ff8a97dd [inlined] std::panic::catch_unwind::h3824cda6fd9d038a(f=<unavailable>) at panic.rs:396 [opt]
    frame #10: 0x0000000108d664bf stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h50d48fa4ff8a97dd [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::had1c36d54f86d88e at mod.rs:473 [opt]
    frame #11: 0x0000000108d6641a stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h50d48fa4ff8a97dd((null)=0x00007f8651904250, (null)=<unavailable>) at function.rs:227 [opt]
    frame #12: 0x00000001098f3a7d stacks-node`std::sys::unix::thread::Thread::new::thread_start::hedb7cc0d930a8f40 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h22f4d58d7c8a821c at boxed.rs:1307:9 [opt]
    frame #13: 0x00000001098f3a77 stacks-node`std::sys::unix::thread::Thread::new::thread_start::hedb7cc0d930a8f40 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::haa812e1889643845 at boxed.rs:1307 [opt]
    frame #14: 0x00000001098f3a6e stacks-node`std::sys::unix::thread::Thread::new::thread_start::hedb7cc0d930a8f40 at thread.rs:71 [opt]
    frame #15: 0x00007fff68e2e109 libsystem_pthread.dylib`_pthread_start + 148
    frame #16: 0x00007fff68e29b8b libsystem_pthread.dylib`thread_start + 15
  thread #3, name = 'chains-coordinator'
    frame #0: 0x00007fff68d6d882 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff68e2e425 libsystem_pthread.dylib`_pthread_cond_wait + 698
    frame #2: 0x00000001094c278f stacks-node`blockstack_lib::chainstate::coordinator::comm::CoordinatorReceivers::wait_on::hba425e7931aa3126 [inlined] std::sys::unix::condvar::Condvar::wait::hda4f2ed95c6070cd(self=<unavailable>, mutex=<unavailable>) at condvar.rs:71:17 [opt]
    frame #3: 0x00000001094c2784 stacks-node`blockstack_lib::chainstate::coordinator::comm::CoordinatorReceivers::wait_on::hba425e7931aa3126 [inlined] std::sys_common::condvar::Condvar::wait::h12340c550f4b6646(self=<unavailable>, mutex=0x00007f8651904a80) at condvar.rs:44 [opt]
    frame #4: 0x00000001094c276c stacks-node`blockstack_lib::chainstate::coordinator::comm::CoordinatorReceivers::wait_on::hba425e7931aa3126 at condvar.rs:188 [opt]
    frame #5: 0x00000001094c276c stacks-node`blockstack_lib::chainstate::coordinator::comm::CoordinatorReceivers::wait_on::hba425e7931aa3126(self=<unavailable>) at comm.rs:122 [opt]
    frame #6: 0x0000000108c0b5ed stacks-node`blockstack_lib::chainstate::coordinator::ChainsCoordinator$LT$T$C$blockstack_lib..chainstate..coordinator..comm..ArcCounterCoordinatorNotices$C$blockstack_lib..chainstate..coordinator..OnChainRewardSetProvider$GT$::run::h53068dea5da6e7c4(chain_state_db=<unavailable>, burnchain=<unavailable>, attachments_tx=<unavailable>, dispatcher=<unavailable>, comms=<unavailable>, atlas_config=<unavailable>) at mod.rs:295:19 [opt]
    frame #7: 0x0000000108d5c659 stacks-node`std::sys_common::backtrace::__rust_begin_short_backtrace::h497736db1d3cc093 [inlined] stacks_node::run_loop::neon::RunLoop::start::_$u7b$$u7b$closure$u7d$$u7d$::h5c47e30aa1bca79a at neon.rs:261:17 [opt]
    frame #8: 0x0000000108d5c5c9 stacks-node`std::sys_common::backtrace::__rust_begin_short_backtrace::h497736db1d3cc093(f=<unavailable>) at backtrace.rs:125 [opt]
    frame #9: 0x0000000108d666f1 stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h5b8ec34011982225 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h44777b5ee157c3e1 at mod.rs:474:17 [opt]
    frame #10: 0x0000000108d666cd stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h5b8ec34011982225 [inlined] _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h83bdd360b1a7a892(self=<unavailable>) at panic.rs:322 [opt]
    frame #11: 0x0000000108d666cd stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h5b8ec34011982225 at panicking.rs:381 [opt]
    frame #12: 0x0000000108d666cd stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h5b8ec34011982225 at panicking.rs:345 [opt]
    frame #13: 0x0000000108d666cd stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h5b8ec34011982225 [inlined] std::panic::catch_unwind::h499dd2a84ae61cad(f=<unavailable>) at panic.rs:396 [opt]
    frame #14: 0x0000000108d666cd stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h5b8ec34011982225 [inlined] std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h2eaeb14f08246660 at mod.rs:473 [opt]
    frame #15: 0x0000000108d6661d stacks-node`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h5b8ec34011982225((null)=0x00007f865300ba00, (null)=<unavailable>) at function.rs:227 [opt]
    frame #16: 0x00000001098f3a7d stacks-node`std::sys::unix::thread::Thread::new::thread_start::hedb7cc0d930a8f40 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h22f4d58d7c8a821c at boxed.rs:1307:9 [opt]
    frame #17: 0x00000001098f3a77 stacks-node`std::sys::unix::thread::Thread::new::thread_start::hedb7cc0d930a8f40 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::haa812e1889643845 at boxed.rs:1307 [opt]
    frame #18: 0x00000001098f3a6e stacks-node`std::sys::unix::thread::Thread::new::thread_start::hedb7cc0d930a8f40 at thread.rs:71 [opt]
    frame #19: 0x00007fff68e2e109 libsystem_pthread.dylib`_pthread_start + 148
    frame #20: 0x00007fff68e29b8b libsystem_pthread.dylib`thread_start + 15
(lldb)
stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 2 years ago

This issue has been automatically closed. Please reopen if needed.

blockstack-devops commented 3 days ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.