search

ZcashFoundation / zebra

Zcash - Financial Privacy in Rust 🦓
https://zfnd.org/zebra/
Apache License 2.0
410 stars 102 forks source link

bug: Crash: Sprout note commitment tree must exist if there is a finalized tip #7961

Closed hhanh00 closed 11 months ago

hhanh00 commented 11 months ago

What happened?

I expected to see this happen: zebra starts

Instead, this happened: Zebra is in restart crash loop (it is running under systemd).

0|zebrad | The application panicked (crashed). 0|zebrad | Message: Sprout note commitment tree must exist if there is a finalized tip 0|zebrad | Location: zebra-state/src/service/finalized_state/zebra_db/shielded.rs:119 0|zebrad | Metadata: 0|zebrad | version: 1.4.0+14.gc3b11e3 0|zebrad | Zcash network: Mainnet 0|zebrad | running state version: 25.3.0 0|zebrad | initial disk state version: 25.2.2 0|zebrad | features: default,getblocktemplate_rpcs,howudoin,indicatif,progress_bar,release_max_level_info 0|zebrad | branch: main 0|zebrad | git commit: c3b11e3 0|zebrad | commit timestamp: 2023-11-16T21:22:34.000000000Z 0|zebrad | target triple: x86_64-unknown-linux-gnu 0|zebrad | rust compiler: 1.72.0 0|zebrad | rust release date: 2023-08-23 0|zebrad | optimization level: 3 0|zebrad | debug checks: false

What were you doing when the issue happened?

No response

Zebra logs

No response

Zebra Version

No response

Which operating systems does the issue happen on?

OS details

No response

Additional information

No response

teor2345 commented 11 months ago

Hi, thanks for reporting this bug!

It looks like you've run into a state bug that is fixed during the first launch of Zebra after upgrading to version 1.4.0. Unfortunately the bug is a race condition, so it can happen during the first launch. The upgrade itself can also cause a similar race. We have to panic if this happens, because if we don't, Zebra could validate the wrong block. (Which is why we needed to fix the bug.)

Try running Zebra again, and hopefully this time the upgrade will win.

You can see more details in PR #7392.

If this continues to happen after 5 or more launches, let us know, and we can change how that code works. But we'll need Zebra's full logs to work out which of the races is actually happening.

hhanh00 commented 11 months ago

Zebra is in restart crash loop (it is running under systemd). Is there a way to recover without having to redownload the data?

teor2345 commented 11 months ago

Zebra is in restart crash loop (it is running under systemd). Is there a way to recover without having to redownload the data?

I don't know without seeing the full logs. Please send us the logs from starting Zebra to it crashing. Ideally the logs for the first crash, but any crash would help.

hhanh00 commented 11 months ago

Where do you get the full logs?

teor2345 commented 11 months ago

Where do you get the full logs?

Using journalctl, the instructions are here: https://unix.stackexchange.com/questions/225401/how-to-see-full-log-from-systemctl-status-service/225407#225407

Or if you've enabled progress bars or the log file in zebrad.toml, it could be at $HOME/.local/state/zebrad.log, or perhaps $XDG_CONFIG_HOME - whatever your systemd LogsDirectory is. That would be $HOME/.config/zebrad.log.

The default depends on your systemd version: https://github.com/systemd/systemd/issues/25739

hhanh00 commented 11 months ago

logfile is not enabled in zebrad.toml. Actually, i am using pm2 (not systemd). Only output I found is from std_out.

The application panicked (crashed).
Message:  Zebra refuses to run if the release date is older than 112 days. 
Release name: 1.1.0, Estimated release height: 2165000 
Hint: Download and install the latest Zebra release from: https://github.com/ZcashFoundation/zebra/releases/latest
Location: zebrad/src/components/sync/end_of_support.rs:81

Metadata:
version: 1.1.0+12.g2c2fa63
Zcash network: Mainnet
running state version: 25.0.2
initial disk state version: 25.0.2
features: default,release_max_level_info
branch: main
git commit: 2c2fa63
commit timestamp: 2023-08-04T10:51:15.000000000Z
target triple: x86_64-unknown-linux-gnu
rust compiler: 1.70.0-nightly
rust release date: 2023-03-27
optimization level: 3
debug checks: false

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   0: zebrad::application:: with zebrad="2c2fa63" net="Main"
      at zebrad/src/application.rs:460

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Thank you for running a mainnet zebrad 1.4.0+14.gc3b11e3 node!
You're helping to strengthen the network and contributing to a social good :)
The application panicked (crashed).
Message:  Sprout note commitment tree must exist if there is a finalized tip
Location: zebra-state/src/service/finalized_state/zebra_db/shielded.rs:119

Metadata:
version: 1.4.0+14.gc3b11e3
Zcash network: Mainnet
running state version: 25.3.0
initial disk state version: 25.0.2
features: default,getblocktemplate_rpcs,howudoin,indicatif,progress_bar,release_max_level_info
branch: main
git commit: c3b11e3
commit timestamp: 2023-11-16T21:22:34.000000000Z
target triple: x86_64-unknown-linux-gnu
rust compiler: 1.72.0
rust release date: 2023-08-23
optimization level: 3
debug checks: false

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   0: zebra_state::service::finalized_state::disk_format::upgrade::cache_genesis_roots::run with _initial_tip_height=Height(2299298)
      at zebra-state/src/service/finalized_state/disk_format/upgrade/cache_genesis_roots.rs:22

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

Consider reporting this error using this URL: https://github.com/ZcashFoundation/zebra/issues/new?title=%3Cautogenerated-issue%3E&body=%23%23+Error%0A%60%60%60%0ASprout+note+commitment+tree+must+exist+if+there+is+a+finalized+tip%0A%60%60%60%0A%0A%23%23+Metadata%0A%7Ckey%7Cvalue%7C%0A%7C--%7C--%7C%0A%7C**version**%7C1.4.0%2B14.gc3b11e3%7C%0A%7C**Zcash+network**%7CMainnet%7C%0A%7C**running+state+version**%7C25.3.0%7C%0A%7C**initial+disk+state+version**%7C25.0.2%7C%0A%7C**features**%7Cdefault%2Cgetblocktemplate_rpcs%2Chowudoin%2Cindicatif%2Cprogress_bar%2Crelease_max_level_info%7C%0A%7C**branch**%7Cmain%7C%0A%7C**git+commit**%7Cc3b11e3%7C%0A%7C**commit+timestamp**%7C2023-11-16T21%3A22%3A34.000000000Z%7C%0A%7C**target+triple**%7Cx86_64-unknown-linux-gnu%7C%0A%7C**rust+compiler**%7C1.72.0%7C%0A%7C**rust+release+date**%7C2023-08-23%7C%0A%7C**optimization+level**%7C3%7C%0A%7C**debug+checks**%7Cfalse%7C%0A%7C**location**%7Czebra-state%2Fsrc%2Fservice%2Ffinalized_state%2Fzebra_db%2Fshielded.rs%3A119%3A21%7C%0A%0A%0A%23%23+SpanTrace%0A%0A%3Cdetails%3E%0A%0A%60%60%60%0ASpanTrace%3A%0A+++0%3A+zebra_state%3A%3Aservice%3A%3Afinalized_state%3A%3Adisk_format%3A%3Aupgrade%3A%3Acache_genesis_roots%3A%3Arun%0A+++++++++++with+_initial_tip_height%3DHeight%282299298%29%0A+++++++++++++at+zebra-state%2Fsrc%2Fservice%2Ffinalized_state%2Fdisk_format%2Fupgrade%2Fcache_genesis_roots.rs%3A22%0A%60%60%60%0A%3C%2Fdetails%3E
Thank you for running a mainnet zebrad 1.4.0+14.gc3b11e3 node!
You're helping to strengthen the network and contributing to a social good :)
The application panicked (crashed).

But it is still crashing, so it should be easy to get other logs if needed.

Between these runs, I rebuilt the latest HEAD

teor2345 commented 11 months ago

Where do you get the full logs?

Using journalctl, the instructions are here: unix.stackexchange.com/questions/225401/how-to-see-full-log-from-systemctl-status-service/225407#225407

Or if you've enabled progress bars or the log file in zebrad.toml, it could be at $HOME/.local/state/zebrad.log, or perhaps $XDG_CONFIG_HOME - whatever your systemd LogsDirectory is. That would be $HOME/.config/zebrad.log.

The default depends on your systemd version: systemd/systemd#25739

If you recompile with cargo build --release --features journald, then this command should work.

Or you can update the systemd unit file to put the logs in the journal using StandardOutput=journal.

hhanh00 commented 11 months ago 
Nov 20 17:47:20 archlinux systemd[1]: Started Process Core Dump (PID 1929845/UID 0).
Nov 20 17:47:20 archlinux systemd-coredump[1929846]: Removed old coredump core.zebrad.1000.ed6eaf7ff0d2486ab421de83990352fe.1570933.1700351179000000.zst.
Nov 20 17:47:22 archlinux systemd-coredump[1929846]: Process 1929755 (zebrad) of user 1000 dumped core.

                                                     Stack trace of thread 1929839:
                                                     #0  0x00007f93161bf83c n/a (libc.so.6 + 0x8e83c)
                                                     #1  0x00007f931616f668 raise (libc.so.6 + 0x3e668)
                                                     #2  0x00007f93161574b8 abort (libc.so.6 + 0x264b8)
                                                     #3  0x0000559bdf9738f7 _ZN11panic_abort18__rust_start_panic5abort17haafe9baa1f76537cE.llvm.4155379323309814477 (zebrad + 0x4908f7)
                                                     #4  0x0000559bdfadf996 rust_panic (zebrad + 0x5fc996)
                                                     #5  0x0000559bdfadf8d1 _ZN3std9panicking20rust_panic_with_hook17hceaf38da6d9db792E (zebrad + 0x5fc8d1)
                                                     #6  0x0000559bdfadf644 _ZN3std9panicking19begin_panic_handler28_$u7b$$u7b$closure$u7d$$u7d$17h2bce3ed2516af7dfE (zebrad + 0x5fc644)
                                                     #7  0x0000559bdfade116 _ZN3std10sys_common9backtrace26__rust_end_short_backtrace17h090f3faf8f98a395E.llvm.16160911627453087270 (zebrad + 0x5fb116)
                                                     #8  0x0000559bdfadf3d2 rust_begin_unwind (zebrad + 0x5fc3d2)
                                                     #9  0x0000559bdf60d2d3 _ZN4core9panicking9panic_fmt17h4ec8274704d163a3E (zebrad + 0x12a2d3)
                                                     #10 0x0000559bdf60d293 _ZN4core6option13expect_failed17h3de37afca26e8e59E (zebrad + 0x12a293)
                                                     #11 0x0000559bdfd64515 _ZN11zebra_state7service15finalized_state8zebra_db8shielded74_$LT$impl$u20$zebra_state..service..finalized_state..zebra_db..ZebraDb$GT$19sprout_tree_for_tip17haf1d2644515d0914E (zebrad + 0x881515)
                                                     #12 0x0000559bdfd07e0b _ZN11zebra_state7service15finalized_state11disk_format7upgrade19cache_genesis_roots3run17he9806cfd402a4cf1E (zebrad + 0x824e0b)
                                                     #13 0x0000559bdfd36c64 _ZN11zebra_state7service15finalized_state11disk_format7upgrade14DbFormatChange26run_format_change_or_check17h853939f397e9e950E (zebrad + 0x853c64)
                                                     #14 0x0000559bdfd23030 _ZN7tracing4span4Span8in_scope17h67c7cda454ead101E (zebrad + 0x840030)
                                                     #15 0x0000559bdfdc5524 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hea690a51a66132d8E (zebrad + 0x8e2524)
                                                     #16 0x0000559bdfd9e93d _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h863ca25a3cd3d300E (zebrad + 0x8bb93d)
                                                     #17 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #18 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #19 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929757:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929758:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929761:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929765:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929772:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929766:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929767:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929764:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929769:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929771:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929775:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929774:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929789:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929759:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929777:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929760:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929756:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929773:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929755:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfafd82e _ZN5tokio7runtime4park5Inner4park17h7bb13af9de3c5984E.llvm.13022031749620168941 (zebrad + 0x61a82e)
                                                     #3  0x0000559bdfe6988c _ZN5tokio7runtime4park16CachedParkThread8block_on17h9b7754b57230f95fE (zebrad + 0x98688c)
                                                     #4  0x0000559be00497aa _ZN5tokio7runtime7context7runtime13enter_runtime17hbb4bfc98f0603385E (zebrad + 0xb667aa)
                                                     #5  0x0000559be008d419 _ZN90_$LT$tokio..runtime..runtime..Runtime$u20$as$u20$zebrad..components..tokio..RuntimeRun$GT$3run17h2665fd82fbc598adE (zebrad + 0xbaa419)
                                                     #6  0x0000559bdfe91a7a _ZN87_$LT$zebrad..commands..start..StartCmd$u20$as$u20$abscissa_core..runnable..Runnable$GT$3run17h9c5321b21b6ea2f6E (zebrad + 0x9aea7a)
                                                     #7  0x0000559bdfe5343a _ZN13abscissa_core11application11Application3run17h71f7a313508587b8E.llvm.5963163806290902465 (zebrad + 0x97043a)
                                                     #8  0x0000559bdfe7b191 _ZN6zebrad11application4boot17h8f522d23b5f250cfE (zebrad + 0x998191)
                                                     #9  0x0000559be010ca2e _ZN6zebrad4main17hc2bf1bb79ac5f5cdE (zebrad + 0xc29a2e)
                                                     #10 0x0000559be010c9e3 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h0a316d9edd1c643dE (zebrad + 0xc299e3)
                                                     #11 0x0000559be010c9f9 _ZN3std2rt10lang_start28_$u7b$$u7b$closure$u7d$$u7d$17h68de712c6c7c1f7eE (zebrad + 0xc299f9)
                                                     #12 0x0000559bdfad2ca5 _ZN3std2rt19lang_start_internal17h5f408694586c2a05E (zebrad + 0x5efca5)
                                                     #13 0x0000559be010ca55 main (zebrad + 0xc29a55)
                                                     #14 0x00007f9316158cd0 n/a (libc.so.6 + 0x27cd0)
                                                     #15 0x00007f9316158d8a __libc_start_main (libc.so.6 + 0x27d8a)
                                                     #16 0x0000559bdf6daeb5 _start (zebrad + 0x1f7eb5)

                                                     Stack trace of thread 1929783:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929763:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929786:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929762:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929841:
                                                     #0  0x00007f9316795642 n/a (ld-linux-x86-64.so.2 + 0x7642)
                                                     #1  0x00007f93167966b1 n/a (ld-linux-x86-64.so.2 + 0x86b1)
                                                     #2  0x00007f931679a049 n/a (ld-linux-x86-64.so.2 + 0xc049)
                                                     #3  0x00007f931678f4e1 _dl_catch_exception (ld-linux-x86-64.so.2 + 0x14e1)
                                                     #4  0x00007f931679981a n/a (ld-linux-x86-64.so.2 + 0xb81a)
                                                     #5  0x00007f931678f4e1 _dl_catch_exception (ld-linux-x86-64.so.2 + 0x14e1)
                                                     #6  0x00007f9316799bec n/a (ld-linux-x86-64.so.2 + 0xbbec)
                                                     #7  0x00007f93162858a1 n/a (libc.so.6 + 0x1548a1)
                                                     #8  0x00007f931678f4e1 _dl_catch_exception (ld-linux-x86-64.so.2 + 0x14e1)
                                                     #9  0x00007f931678f603 n/a (ld-linux-x86-64.so.2 + 0x1603)
                                                     #10 0x00007f9316285811 n/a (libc.so.6 + 0x154811)
                                                     #11 0x00007f9316285a4f n/a (libc.so.6 + 0x154a4f)
                                                     #12 0x00007f931626d857 n/a (libc.so.6 + 0x13c857)
                                                     #13 0x00007f931626dc3d n/a (libc.so.6 + 0x13cc3d)
                                                     #14 0x00007f9316213aec getaddrinfo (libc.so.6 + 0xe2aec)
                                                     #15 0x0000559bdfae4c27 _ZN104_$LT$std..sys_common..net..LookupHost$u20$as$u20$core..convert..TryFrom$LT$$LP$$RF$str$C$u16$RP$$GT$$GT$8try_from28_$u7b$$u7b$closure$u7d$$u7d$17h509d796f74cad1a6E (zebrad + 0x601c27)
                                                     #16 0x0000559bdfae4b99 _ZN90_$LT$std..sys_common..net..LookupHost$u20$as$u20$core..convert..TryFrom$LT$$RF$str$GT$$GT$8try_from17h3e5df4df41d9275eE (zebrad + 0x601b99)
                                                     #17 0x0000559bdfadb5b6 _ZN60_$LT$str$u20$as$u20$std..net..socket_addr..ToSocketAddrs$GT$15to_socket_addrs17h8bc6d4e6c43109aeE (zebrad + 0x5f85b6)
                                                     #18 0x0000559bdfb06b60 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17he1083e2633c89dfdE (zebrad + 0x623b60)
                                                     #19 0x0000559bdfaf773d _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17hcaf568c78873d378E (zebrad + 0x61473d)
                                                     #20 0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #21 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #22 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #23 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #24 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #25 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929768:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929778:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929807:
                                                     #0  0x00007f93161ba4ae n/a (libc.so.6 + 0x894ae)
                                                     #1  0x00007f93161bcd40 pthread_cond_wait (libc.so.6 + 0x8bd40)
                                                     #2  0x00007f93164d9e11 __gthread_cond_wait (libstdc++.so.6 + 0xd9e11)
                                                     #3  0x0000559be0481b08 _ZN7rocksdb14ThreadPoolImpl4Impl8BGThreadEm (zebrad + 0xf9eb08)
                                                     #4  0x0000559be0481ee3 _ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv (zebrad + 0xf9eee3)
                                                     #5  0x00007f93164e1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                     #6  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #7  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929780:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdfad3055 _ZN3std6thread4park17h5d9fb3dc7f70c076E (zebrad + 0x5f0055)
                                                     #2  0x0000559bdff0144d _ZN17crossbeam_channel7context7Context10wait_until17hcc75aebaf2d2ee31E (zebrad + 0xa1e44d)
                                                     #3  0x0000559bdff01652 _ZN17crossbeam_channel7context7Context4with28_$u7b$$u7b$closure$u7d$$u7d$17ha0f4342435315ec5E (zebrad + 0xa1e652)
                                                     #4  0x0000559bdff01c9e _ZN17crossbeam_channel7flavors5array16Channel$LT$T$GT$4recv17h9cbee6d93995b276E (zebrad + 0xa1ec9e)
                                                     #5  0x0000559bdfff1897 _ZN17crossbeam_channel7channel17Receiver$LT$T$GT$4recv17h0113fc2bf936b660E (zebrad + 0xb0e897)
                                                     #6  0x0000559bdffd20fe _ZN16tracing_appender6worker15Worker$LT$T$GT$4work17h8a43eb588deaa204E (zebrad + 0xaef0fe)
                                                     #7  0x0000559be0042a10 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h523f511612df1151E (zebrad + 0xb5fa10)
                                                     #8  0x0000559bdfff5db2 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h4d5e2f6d461d41a6E (zebrad + 0xb12db2)
                                                     #9  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #10 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #11 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929776:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929795:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929793:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929804:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929782:
                                                     #0  0x00007f9316241bf6 epoll_wait (libc.so.6 + 0x110bf6)
                                                     #1  0x0000559bdf899c7e _ZN3mio3sys4unix8selector5epoll8Selector6select17h8698e8db8b37de3aE (zebrad + 0x3b6c7e)
                                                     #2  0x0000559bdfb0fb4e _ZN5tokio7runtime2io6driver6Driver4turn17hb56acc2dda54d36fE.llvm.9437723879503481686 (zebrad + 0x62cb4e)
                                                     #3  0x0000559bdfb066f0 _ZN5tokio7runtime4time6Driver19park_thread_timeout17h44885ba67009be28E.llvm.10462101111869551526 (zebrad + 0x6236f0)
                                                     #4  0x0000559bdfb063b8 _ZN5tokio7runtime4time6Driver13park_internal17hf7d183980f4d65fbE.llvm.10462101111869551526 (zebrad + 0x6233b8)
                                                     #5  0x0000559bdfb05b7b _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622b7b)
                                                     #6  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #7  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #8  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #9  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #10 0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #11 0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #12 0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #13 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #14 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #15 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #16 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #17 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929770:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929838:
                                                     #0  0x00007f93161ba4ae n/a (libc.so.6 + 0x894ae)
                                                     #1  0x00007f93161bd055 pthread_cond_timedwait (libc.so.6 + 0x8c055)
                                                     #2  0x0000559be03c752d _ZN7rocksdb4port7CondVar9TimedWaitEm (zebrad + 0xee452d)
                                                     #3  0x0000559be037edda _ZN7rocksdb19InstrumentedCondVar9TimedWaitEm (zebrad + 0xe9bdda)
                                                     #4  0x0000559be0270fd7 _ZN7rocksdb5Timer3RunEv (zebrad + 0xd8dfd7)
                                                     #5  0x00007f93164e1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                     #6  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #7  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929787:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929792:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929785:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929798:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929844:
                                                     #0  0x00007f93161ba700 n/a (libc.so.6 + 0x89700)
                                                     #1  0x00007f93161c0f1a pthread_mutex_lock (libc.so.6 + 0x8ff1a)
                                                     #2  0x00007f93161718d9 __cxa_thread_atexit_impl (libc.so.6 + 0x408d9)
                                                     #3  0x0000559bdfade820 _ZN3std10sys_common11thread_info3set17hf996aea6f40e5f17E (zebrad + 0x5fb820)
                                                     #4  0x0000559bdfb02771 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f771)
                                                     #5  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #6  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #7  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929784:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929801:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929840:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfafd82e _ZN5tokio7runtime4park5Inner4park17h7bb13af9de3c5984E.llvm.13022031749620168941 (zebrad + 0x61a82e)
                                                     #3  0x0000559bdfe123dc _ZN5tokio7runtime4park16CachedParkThread8block_on17h890366201242469cE (zebrad + 0x92f3dc)
                                                     #4  0x0000559bdfd4abf9 _ZN11zebra_state7service5write26write_blocks_from_channels17h8d58f58966a0e8e4E (zebrad + 0x867bf9)
                                                     #5  0x0000559bdfdc52a4 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hb87cdb1af610d487E (zebrad + 0x8e22a4)
                                                     #6  0x0000559bdfd9e774 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h0036161b290d5971E (zebrad + 0x8bb774)
                                                     #7  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #8  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #9  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929781:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929806:
                                                     #0  0x0000559be043733a _ZN7rocksdb25CompactionMergingIterator4NextEv (zebrad + 0xf5433a)
                                                     #1  0x0000559be04ddc20 _ZN7rocksdb18CompactionIterator4NextEv (zebrad + 0xffac20)
                                                     #2  0x0000559be04e5f7f _ZN7rocksdb13CompactionJob25ProcessKeyValueCompactionEPNS_18SubcompactionStateE (zebrad + 0x1002f7f)
                                                     #3  0x0000559be04ed998 _ZN7rocksdb13CompactionJob3RunEv (zebrad + 0x100a998)
                                                     #4  0x0000559be01a96c3 _ZN7rocksdb6DBImpl20BackgroundCompactionEPbPNS_10JobContextEPNS_9LogBufferEPNS0_19PrepickedCompactionENS_3Env8PriorityE (zebrad + 0xcc66c3)
                                                     #5  0x0000559be01aadbf _ZN7rocksdb6DBImpl24BackgroundCallCompactionEPNS0_19PrepickedCompactionENS_3Env8PriorityE (zebrad + 0xcc7dbf)
                                                     #6  0x0000559be01ab4fd _ZN7rocksdb6DBImpl16BGWorkCompactionEPv (zebrad + 0xcc84fd)
                                                     #7  0x0000559be0481cd3 _ZN7rocksdb14ThreadPoolImpl4Impl8BGThreadEm (zebrad + 0xf9ecd3)
                                                     #8  0x0000559be0481ee3 _ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv (zebrad + 0xf9eee3)
                                                     #9  0x00007f93164e1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                     #10 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #11 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929788:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929800:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929842:
                                                     #0  0x00007f93161ba700 n/a (libc.so.6 + 0x89700)
                                                     #1  0x00007f93161c0f1a pthread_mutex_lock (libc.so.6 + 0x8ff1a)
                                                     #2  0x00007f9316799b82 n/a (ld-linux-x86-64.so.2 + 0xbb82)
                                                     #3  0x00007f93162858a1 n/a (libc.so.6 + 0x1548a1)
                                                     #4  0x00007f931678f4e1 _dl_catch_exception (ld-linux-x86-64.so.2 + 0x14e1)
                                                     #5  0x00007f931678f603 n/a (ld-linux-x86-64.so.2 + 0x1603)
                                                     #6  0x00007f9316285811 n/a (libc.so.6 + 0x154811)
                                                     #7  0x00007f9316285a4f n/a (libc.so.6 + 0x154a4f)
                                                     #8  0x00007f931626d857 n/a (libc.so.6 + 0x13c857)
                                                     #9  0x00007f931626dc3d n/a (libc.so.6 + 0x13cc3d)
                                                     #10 0x00007f9316213aec getaddrinfo (libc.so.6 + 0xe2aec)
                                                     #11 0x0000559bdfae4c27 _ZN104_$LT$std..sys_common..net..LookupHost$u20$as$u20$core..convert..TryFrom$LT$$LP$$RF$str$C$u16$RP$$GT$$GT$8try_from28_$u7b$$u7b$closure$u7d$$u7d$17h509d796f74cad1a6E (zebrad + 0x601c27)
                                                     #12 0x0000559bdfae4b99 _ZN90_$LT$std..sys_common..net..LookupHost$u20$as$u20$core..convert..TryFrom$LT$$RF$str$GT$$GT$8try_from17h3e5df4df41d9275eE (zebrad + 0x601b99)
                                                     #13 0x0000559bdfadb5b6 _ZN60_$LT$str$u20$as$u20$std..net..socket_addr..ToSocketAddrs$GT$15to_socket_addrs17h8bc6d4e6c43109aeE (zebrad + 0x5f85b6)
                                                     #14 0x0000559bdfb06b60 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17he1083e2633c89dfdE (zebrad + 0x623b60)
                                                     #15 0x0000559bdfaf773d _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17hcaf568c78873d378E (zebrad + 0x61473d)
                                                     #16 0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #17 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #18 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #19 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #20 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #21 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929803:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929790:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929843:
                                                     #0  0x00007f93161ba700 n/a (libc.so.6 + 0x89700)
                                                     #1  0x00007f93161c0f1a pthread_mutex_lock (libc.so.6 + 0x8ff1a)
                                                     #2  0x00007f9316799b82 n/a (ld-linux-x86-64.so.2 + 0xbb82)
                                                     #3  0x00007f93162858a1 n/a (libc.so.6 + 0x1548a1)
                                                     #4  0x00007f931678f4e1 _dl_catch_exception (ld-linux-x86-64.so.2 + 0x14e1)
                                                     #5  0x00007f931678f603 n/a (ld-linux-x86-64.so.2 + 0x1603)
                                                     #6  0x00007f9316285811 n/a (libc.so.6 + 0x154811)
                                                     #7  0x00007f9316285a4f n/a (libc.so.6 + 0x154a4f)
                                                     #8  0x00007f931626d857 n/a (libc.so.6 + 0x13c857)
                                                     #9  0x00007f931626dc3d n/a (libc.so.6 + 0x13cc3d)
                                                     #10 0x00007f9316213aec getaddrinfo (libc.so.6 + 0xe2aec)
                                                     #11 0x0000559bdfae4c27 _ZN104_$LT$std..sys_common..net..LookupHost$u20$as$u20$core..convert..TryFrom$LT$$LP$$RF$str$C$u16$RP$$GT$$GT$8try_from28_$u7b$$u7b$closure$u7d$$u7d$17h509d796f74cad1a6E (zebrad + 0x601c27)
                                                     #12 0x0000559bdfae4b99 _ZN90_$LT$std..sys_common..net..LookupHost$u20$as$u20$core..convert..TryFrom$LT$$RF$str$GT$$GT$8try_from17h3e5df4df41d9275eE (zebrad + 0x601b99)
                                                     #13 0x0000559bdfadb5b6 _ZN60_$LT$str$u20$as$u20$std..net..socket_addr..ToSocketAddrs$GT$15to_socket_addrs17h8bc6d4e6c43109aeE (zebrad + 0x5f85b6)
                                                     #14 0x0000559bdfb06b60 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17he1083e2633c89dfdE (zebrad + 0x623b60)
                                                     #15 0x0000559bdfaf773d _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17hcaf568c78873d378E (zebrad + 0x61473d)
                                                     #16 0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #17 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #18 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #19 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #20 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #21 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929779:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf621e23 _ZN10rayon_core5sleep5Sleep5sleep17h93cd4187063ef679E (zebrad + 0x13ee23)
                                                     #2  0x0000559bdf6214b3 _ZN10rayon_core8registry12WorkerThread15wait_until_cold17hfebd9f921fa43b9cE (zebrad + 0x13e4b3)
                                                     #3  0x0000559bdf9a19fd _ZN10rayon_core8registry13ThreadBuilder3run17hdadc9216ee5416c9E (zebrad + 0x4be9fd)
                                                     #4  0x0000559bdf9a411a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hd0277fe037e4cc88E.llvm.4102196153362087751 (zebrad + 0x4c111a)
                                                     #5  0x0000559bdf9a56f9 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h597cbb385f39d5b3E (zebrad + 0x4c26f9)
                                                     #6  0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #7  0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #8  0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929791:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929799:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929794:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929796:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929802:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929797:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfb05c2e _ZN5tokio7runtime9scheduler12multi_thread4park6Parker4park17h6b3bd30f83b6cfb5E (zebrad + 0x622c2e)
                                                     #3  0x0000559bdfaffc2b _ZN5tokio7runtime9scheduler12multi_thread6worker7Context12park_timeout17h4feb6e96320acbc9E (zebrad + 0x61cc2b)
                                                     #4  0x0000559bdfaff546 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6133c9dfa825e7f2E (zebrad + 0x61c546)
                                                     #5  0x0000559bdfb0901c _ZN5tokio7runtime7context13set_scheduler17h2f57969b4351419eE (zebrad + 0x62601c)
                                                     #6  0x0000559bdfafe810 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17h821645fdd6da982eE (zebrad + 0x61b810)
                                                     #7  0x0000559bdfb06851 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17h800bda3aca717f94E (zebrad + 0x623851)
                                                     #8  0x0000559bdfaf754a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17haf0bc10e4a747f0fE (zebrad + 0x61454a)
                                                     #9  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #10 0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #11 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #12 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #13 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #14 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)

                                                     Stack trace of thread 1929805:
                                                     #0  0x00007f931623f73d syscall (libc.so.6 + 0x10e73d)
                                                     #1  0x0000559bdf973e09 _ZN11parking_lot7condvar7Condvar19wait_until_internal17he60cefdaf3195a58E (zebrad + 0x490e09)
                                                     #2  0x0000559bdfafd82e _ZN5tokio7runtime4park5Inner4park17h7bb13af9de3c5984E.llvm.13022031749620168941 (zebrad + 0x61a82e)
                                                     #3  0x0000559bdfc8bedc _ZN5tokio7runtime4park16CachedParkThread8block_on17ha99658788bf0ae7fE (zebrad + 0x7a8edc)
                                                     #4  0x0000559bdfc5c256 _ZN7tracing4span4Span8in_scope17h1a281aa4aef04731E (zebrad + 0x779256)
                                                     #5  0x0000559bdfca440b _ZN102_$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$4poll17h030214e5acc6bd0eE (zebrad + 0x7c140b)
                                                     #6  0x0000559bdfc90bd1 _ZN115_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$9call_once17h66d075832b32f80eE (zebrad + 0x7adbd1)
                                                     #7  0x0000559bdfc5b00d _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17hff966616f0ba277aE (zebrad + 0x77800d)
                                                     #8  0x0000559bdfb0bfb5 _ZN5tokio7runtime8blocking4pool5Inner3run17h040684a3fd8cb8bbE (zebrad + 0x628fb5)
                                                     #9  0x0000559bdfafd26a _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5ed2ef0f3ca0b16eE (zebrad + 0x61a26a)
                                                     #10 0x0000559bdfb02779 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h6491d5df12e2afb8E (zebrad + 0x61f779)
                                                     #11 0x0000559bdfae2ef5 _ZN3std3sys4unix6thread6Thread3new12thread_start17h04c8e9c7d83d3bd5E (zebrad + 0x5ffef5)
                                                     #12 0x00007f93161bd9eb n/a (libc.so.6 + 0x8c9eb)
                                                     #13 0x00007f93162417cc n/a (libc.so.6 + 0x1107cc)
                                                     ELF object binary architecture: AMD x86-64
Nov 20 17:47:22 archlinux systemd[1]: systemd-coredump@48190-1929845-0.service: Deactivated successfully.
Nov 20 17:47:22 archlinux systemd[1]: systemd-coredump@48190-1929845-0.service: Consumed 1.296s CPU time.
teor2345 commented 11 months ago

The crash log might be enough for us to work with, but it would still be helpful to have Zebra's info-level logs before the crash. That will tell us which order the upgrade and other accesses are happening in.

hhanh00 commented 11 months ago 
Thank you for running a mainnet zebrad 1.4.0 node!
You're helping to strengthen the network and contributing to a social good :)
2023-11-21T03:04:21.897396Z  INFO zebrad::components::tracing::component: started tracing component filter="info" TRACING_STATIC_MAX_LEVEL=LevelFilter::INFO LOG_STATIC_MAX_LEVEL=Info
2023-11-21T03:04:21.897411Z  INFO zebrad::components::tracing::component: set 'tracing.progress_bar ="summary"' in zebrad.toml to activate progress bars
2023-11-21T03:04:21.897413Z  INFO zebrad::application: Diagnostic Metadata:
version: 1.4.0
Zcash network: Mainnet
running state version: 25.3.0
initial disk state version: 25.2.2
features: default,getblocktemplate_rpcs,howudoin,indicatif,journald,progress_bar,release_max_level_info,tracing_journald
branch: HEAD
git commit: 477da1a
commit timestamp: 2023-11-06T12:51:41.000000000Z
target triple: x86_64-unknown-linux-gnu
rust compiler: 1.72.0
rust release date: 2023-08-23
optimization level: 3
debug checks: false
2023-11-21T03:04:21.897415Z  INFO zebrad::application: loaded zebrad config config_path=Some("/home/hanh/.config/zebrad.toml") config=ZebradConfig { consensus: Config { checkpoint_sync: true }, metrics: Config { endpoint_addr: None }, network: Config { listen_addr: 0.0.0.0:8233, network: Mainnet, initial_mainnet_peers: {"dnsseed.z.cash:8233", "dnsseed.str4d.xyz:8233", "mainnet.seeder.zfnd.org:8233", "mainnet.is.yolo.money:8233"}, initial_testnet_peers: {"dnsseed.testnet.z.cash:18233", "testnet.seeder.zfnd.org:18233", "testnet.is.yolo.money:18233"}, cache_dir: IsEnabled(true), peerset_initial_target_size: 25, crawl_new_peer_interval: 61s, max_connections_per_ip: 1 }, state: Config { cache_dir: "/ext/hanh/.cache/zebra", ephemeral: false, delete_old_database: true, debug_stop_at_height: None, debug_validity_check_interval: None }, tracing: Config { inner: InnerConfig { use_color: true, force_use_color: false, filter: None, buffer_limit: 128000, endpoint_addr: None, flamegraph: None, progress_bar: None, log_file: None, use_journald: false } }, sync: Config { download_concurrency_limit: 50, checkpoint_verify_concurrency_limit: 1000, full_verify_concurrency_limit: 20, parallel_cpu_threads: 0 }, mempool: Config { tx_cost_limit: 80000000, eviction_memory_time: 3600s, debug_enable_at_height: None }, rpc: Config { listen_addr: Some(127.0.0.1:8232), parallel_cpu_threads: 0, debug_force_finished_sync: false }, mining: Config { miner_address: None, extra_coinbase_data: None, debug_like_zcashd: true } }
2023-11-21T03:04:21.897469Z  INFO {zebrad="477da1a" net="Main"}: zebrad::application: initialized rayon thread pool for CPU-bound tasks num_threads=24
2023-11-21T03:04:21.898992Z  INFO {zebrad="477da1a" net="Main"}: zebrad::commands::start: Starting zebrad
2023-11-21T03:04:21.899019Z  INFO {zebrad="477da1a" net="Main"}: zebrad::commands::start: initializing node state
2023-11-21T03:04:21.902706Z  INFO {zebrad="477da1a" net="Main"}: zebrad::commands::start: opening database, this may take a few minutes
2023-11-21T03:04:21.902838Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: trying to open older database format: launching upgrade task running_version=25.3.0 disk_version=25.2.2
2023-11-21T03:04:21.902941Z  INFO zebra_state::service::finalized_state::disk_db: the open file limit is high enough for Zebra current_limit=1024 min_limit=512 ideal_limit=1024
2023-11-21T03:04:22.466030Z  INFO zebra_state::service::finalized_state::disk_db: Opened Zebra state cache at /ext/hanh/.cache/zebra/state/v25/mainnet
2023-11-21T03:04:22.467992Z  INFO zebra_state::service::finalized_state: loaded Zebra state cache tip=Some((Height(2301194), block::Hash("00000000009f26c58b5a8328a22c8d9a691dd70d0e4fc1fef852c6e2edbb109e")))
The application panicked (crashed).
Message:  Sprout note commitment tree must exist if there is a finalized tip
Location: zebra-state/src/service/finalized_state/zebra_db/shielded.rs:119

Metadata:
version: 1.4.0
Zcash network: Mainnet
running state version: 25.3.0
initial disk state version: 25.2.2
features: default,getblocktemplate_rpcs,howudoin,indicatif,journald,progress_bar,release_max_level_info,tracing_journald
branch: HEAD
git commit: 477da1a
commit timestamp: 2023-11-06T12:51:41.000000000Z
target triple: x86_64-unknown-linux-gnu
rust compiler: 1.72.0
rust release date: 2023-08-23
optimization level: 3
debug checks: false

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

2023-11-21T03:04:22.468830Z  INFO zebra_state::service: starting legacy chain check
   0: zebra_state::service::finalized_state::disk_format::upgrade::cache_genesis_roots::run with _initial_tip_height=Height(2301194)
      at zebra-state/src/service/finalized_state/disk_format/upgrade/cache_genesis_roots.rs:22

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.2023-11-21T03:04:22.469194Z  INFO zebra_state::service: cached state consensus branch is valid: no legacy chain found
2023-11-21T03:04:22.469253Z  INFO {zebrad="477da1a" net="Main"}: zebrad::commands::start: initializing network
2023-11-21T03:04:22.469279Z  INFO {zebrad="477da1a" net="Main"}:open_listener{addr=0.0.0.0:8233}: zebra_network::peer_set::initialize: Trying to open Zcash protocol endpoint at 0.0.0.0:8233...
2023-11-21T03:04:22.469319Z  INFO {zebrad="477da1a" net="Main"}:open_listener{addr=0.0.0.0:8233}: zebra_network::peer_set::initialize: Opened Zcash protocol endpoint at 0.0.0.0:8233
2023-11-21T03:04:22.469363Z  INFO {zebrad="477da1a" net="Main"}: zebra_network::address_book_updater: starting the address book updater

Consider reporting this error using this URL: https://github.com/ZcashFoundation/zebra/issues/new?title=%3Cautogenerated-issue%3E&body=%23%23+Error%0A%60%60%60%0ASprout+note+commitment+tree+must+exist+if+there+is+a+finalized+tip%0A%60%60%60%0A%0A%23%23+Metadata%0A%7Ckey%7Cvalue%7C%0A%7C--%7C--%7C%0A%7C**version**%7C1.4.0%7C%0A%7C**Zcash+network**%7CMainnet%7C%0A%7C**running+state+version**%7C25.3.0%7C%0A%7C**initial+disk+state+version**%7C25.2.2%7C%0A%7C**features**%7Cdefault%2Cgetblocktemplate_rpcs%2Chowudoin%2Cindicatif%2Cjournald%2Cprogress_bar%2Crelease_max_level_info%2Ctracing_journald%7C%0A%7C**branch**%7CHEAD%7C%0A%7C**git+commit**%7C477da1a%7C%0A%7C**commit+timestamp**%7C2023-11-06T12%3A51%3A41.000000000Z%7C%0A%7C**target+triple**%7Cx86_64-unknown-linux-gnu%7C%0A%7C**rust+compiler**%7C1.72.0%7C%0A%7C**rust+release+date**%7C2023-08-23%7C%0A%7C**optimization+level**%7C3%7C%0A%7C**debug+checks**%7Cfalse%7C%0A%7C**location**%7Czebra-state%2Fsrc%2Fservice%2Ffinalized_state%2Fzebra_db%2Fshielded.rs%3A119%3A21%7C%0A%0A%0A%23%23+SpanTrace%0A%0A%3Cdetails%3E%0A%0A%60%60%60%0ASpanTrace%3A%0A+++0%3A+zebra_state%3A%3Aservice%3A%3Afinalized_state%3A%3Adisk_format%3A%3Aupgrade%3A%3Acache_genesis_roots%3A%3Arun%0A+++++++++++with+%1B%5B3m_initial_tip_height%1B%5B0m%1B%5B2m%3D%1B%5B0mHeight%282301194%29%0A+++++++++++++at+zebra-state%2Fsrc%2Fservice%2Ffinalized_state%2Fdisk_format%2Fupgrade%2Fcache_genesis_roots.rs%3A22%0A%60%60%60%0A%3C%2Fdetails%3E
Aborted (core dumped)
teor2345 commented 11 months ago

Thanks for the logs! Here's my initial analysis for the Zebra developers:

The line that panics is called from the genesis upgrade, which is the first 25.3.0 upgrade code that runs. So we should be operating on a 25.2.2 format database.

https://github.com/ZcashFoundation/zebra/blob/3be22b2cdaa59cd1c028432764f718a3cbef370d/zebra-state/src/service/finalized_state/disk_format/upgrade/cache_genesis_roots.rs#L29

The line that panics is: https://github.com/ZcashFoundation/zebra/blob/3be22b2cdaa59cd1c028432764f718a3cbef370d/zebra-state/src/service/finalized_state/zebra_db/shielded.rs#L119

The only way it can panic is if there is no sprout tree entry for () (25.3.0 format), and no sprout entry for the tip height (25.2.2 format). This shouldn't be possible, because the only ways that sprout trees are modified is:

  1. When a new block is finalized, in the database transaction that also updates the tip height.
  2. During the 25.3.0 upgrade, in a database transaction that changes the key to ().

So this could be caused by:

  1. Database corruption on disk.
  2. A bug in Zebra's production release database reading or writing code, but that's unlikely because other instances would also be failing. The Zebra team can test this by doing an upgrade from 25.2.2 to 25.3.0 ourselves. (v1.3.0 to v1.4.0.)
  3. Using unsupported Zebra versions with bugs in this code (including some previous main branch commits).
  4. Downgrading to older or unsupported Zebra versions after an upgrade.

Unfortunately this tree is essential to verifying blocks correctly, and doing the format upgrade correctly. So there's no quick fix here if it's missing. It might be possible to copy the tree data across from sprout_anchors to sprout_note_commitment_tree, because we know your state height from your logs. https://github.com/ZcashFoundation/zebra/blob/main/book/src/dev/state-db-upgrades.md#current-state-database-format

But that's an unusual thing to do, so I'd like to discuss it with the team. And before that, we should see if we can reproduce this panic ourselves by upgrading from v1.3.0 to v1.4.0.

For @hhanh00:

It would be great to see what's actually in your database causing this problem, I'll post a command that dumps the data in my next comment.

teor2345 commented 11 months ago

If you run this command, it will dump the data that's causing the bug. Feel free to post it here, it's public data.

The ldb command is installed by the rocksdb, rocksdb-dev, or rocksdb.tools packages, depending on your OS and distribution.

This command opens the database as read-only, so it is safe to run.

ldb --try_load_options --db=/ext/hanh/.cache/zebra/state/v25/mainnet --secondary_path= --column_family=sprout_note_commitment_tree --hex scan

For example, on my 25.3.0 testnet instance, I get:

$ ldb --try_load_options --db=/home/dev/.cache/zebra-custom/state/v25/testnet --secondary_path --column_family=sprout_note_commitment_tree --hex scan
0x : 0x01FCF1500100011271C9D146631782AFED036CB6F860ADCD57092E6CDADF61346E635216FFBEDE49D64AC5C40C975381BBB5F2D25B91DC3BC8BECCAC133FF657D5A9F4A629E54F07E59F7EFCF1CF89DE9A81A31767609D420AE4E93B020A3E935FEF52F9C689A9ECF32C39E6E6ED9EBA40BB5C8F19BE51CF857CE6E93B4E501DED57C3EAB0EEF648350935B554E34A1D3BD7CF07291C74D60B950D875BE6911785B62BDB2FBB51992C6FBE790B7089B1214810B82BD16296D5B6CE4601D3DCA75A52DFC15463C0520F8859CDC42B826867502DB1A3642E2A3480C8FB2CB789BAADB71DA2B5E3146893E6594CE4448C3D06F25F3F1E5AACF5CCAD9C5292A3C03EA50347C03B064C864A16EE55BB6CAB4005ABF9065EA418C4E8F8E5D76E7D2548BE24E1C8926C2F8001625F0A5B07FB9FCAB07968A24E5F8E0B3656989C71B27E2ECEC1165BCBA2C4AA

The exact output depends on the state version and block height.

teor2345 commented 11 months ago

And before that, we should see if we can reproduce this panic ourselves by upgrading from v1.3.0 to v1.4.0.

@hhanh00 what Zebra version were you using before you upgraded to v1.4.0?

teor2345 commented 11 months ago

For the Zebra developers, here are the steps I used to test upgrading v1.3.0 to v1.4.0:

Sync some blocks with 1.3.0:

$ cargo install zebrad --version 1.3.0 --locked
# action: created a custom config with a different state path
$ /home/dev/.cargo/bin/zebrad ~/.config/zebra/zebrad-mainnet-alt.toml
# action: waited until some blocks were committed
2023-11-21T04:21:01.698890Z  INFO {net="Main"}: zebrad::components::sync::progress: estimated progress to chain tip sync_percent=0.103% current_height=Height(2400) network_upgrade=BeforeOverwinter remaining_sync_blocks=2316572 time_since_last_state_block=0s
# note: this is the 1.3.0 and earlier format, with a height key
$ ldb --try_load_options --db=/home/dev/.cache/zebra-custom-alt/state/v25/mainnet --secondary_path= --column_family=sprout_note_commitment_tree --hex scan
0x000960 : 0x01FBD924017C46FFED84992822BC6462420147906BA8DEE83ED1D0BF897E46F28CBC4FCCA8461E056128EE0DB9204C54EED1C3CA07639FE403D820E02853AA500B325FDBF8067EE818DA10812E9843EC5C8496ABD25F451ABE3682176E94A816B46BFFA70F3B784280C23ED09FD9F96F3AD03CB5D12BFEAAB50AE7DBACE9818AC6974D0FE6AB272AEB0167D1B16936BD8F4D0E9B3E7F59A5F5DB8C8B33E31E07032933AC57620A772C2EC3788A36183D3A08CAA97D741528F1E6EAC4C6AC94D690DF9009B6A8AD423AE4A0B12DD16D736EEC64DE4ADB50EA9BBB5FD9800069FF2285A7A184584EEA8BE5D18ECA477F59F690D191D9D08ECC9E98D4289E1887917B1BE8F0B4E0019835F41D8ABAB013B00ACD2B52160D38E5A4E58E5E8C97E8A35648A063BA4966

Upgrade to 1.4.0:

$ cargo install zebrad --version 1.4.0 --locked
$ /home/dev/.cargo/bin/zebrad ~/.config/zebra/zebrad-mainnet-alt.toml
# action: waited until the upgrade was complete
2023-11-21T04:26:02.309591Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: trying to open older database format: launching upgrade task running_version=25.3.0 disk_version=25.2.2
...
2023-11-21T04:26:02.770916Z  INFO zebra_state::service::finalized_state::disk_db: Opened Zebra state cache at /home/dev/.cache/zebra-custom-alt/state/v25/mainnet
2023-11-21T04:26:02.771840Z  INFO zebra_state::service::finalized_state: loaded Zebra state cache tip=Some((Height(2400), block::Hash("000000048273afbaf67cc953b5b0c4fc41faf985b815fcbab737689ae9a45021")))
2023-11-21T04:26:02.772411Z  INFO zebra_state::service: starting legacy chain check
2023-11-21T04:26:02.772544Z  INFO zebra_state::service: cached state consensus branch is valid: no legacy chain found
...
2023-11-21T04:26:02.775837Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: marked database format as upgraded running_version=25.3.0 disk_version=25.2.2 format_upgrade_version=25.3.0
2023-11-21T04:26:02.775847Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: Zebra automatically upgraded the database format to: newer_running_version=25.3.0
2023-11-21T04:26:02.777840Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: database format is valid running_version=25.3.0 inital_disk_version=25.2.2
# note: this is the 1.4.0 and later format, with an empty key
$ ldb --try_load_options --db=/home/dev/.cache/zebra-custom-alt/state/v25/mainnet --secondary_path= --column_family=sprout_note_commitment_tree --hex scan
0x : 0x01FBD924017C46FFED84992822BC6462420147906BA8DEE83ED1D0BF897E46F28CBC4FCCA8461E056128EE0DB9204C54EED1C3CA07639FE403D820E02853AA500B325FDBF8067EE818DA10812E9843EC5C8496ABD25F451ABE3682176E94A816B46BFFA70F3B784280C23ED09FD9F96F3AD03CB5D12BFEAAB50AE7DBACE9818AC6974D0FE6AB272AEB0167D1B16936BD8F4D0E9B3E7F59A5F5DB8C8B33E31E07032933AC57620A772C2EC3788A36183D3A08CAA97D741528F1E6EAC4C6AC94D690DF9009B6A8AD423AE4A0B12DD16D736EEC64DE4ADB50EA9BBB5FD9800069FF2285A7A184584EEA8BE5D18ECA477F59F690D191D9D08ECC9E98D4289E1887917B1BE8F0B4E0019835F41D8ABAB013B00ACD2B52160D38E5A4E58E5E8C97E8A35648A063BA4966

I wasn't able to reproduce the panic, starting 1.4.0 again also worked.

hhanh00 commented 11 months ago

It is up there in the logs.

1.1.0+12.g2c2fa63

teor2345 commented 11 months ago

It is up there in the logs.

1.1.0+12.g2c2fa63

Thanks! I just needed to confirm there was nothing in between. I will see if I can reproduce this bug with a v1.1.0 to v1.4.0 upgrade.

teor2345 commented 11 months ago

For the Zebra developers, here are the steps I used to test upgrading v1.1.0 to v1.4.0:

Sync some blocks with 1.1.0:

$ rm -rf /home/dev/.cache/zebra-custom-alt/state
$ cargo install zebrad --version 1.1.0 --locked
$ /home/dev/.cargo/bin/zebrad ~/.config/zebra/zebrad-mainnet-alt.toml
# action: waited until some blocks were committed
2023-11-21T04:39:25.602190Z  INFO {net="Main"}: zebrad::components::sync::progress: estimated progress to chain tip sync_percent=0.017% current_height=Height(400) network_upgrade=BeforeOverwinter remaining_sync_blocks=2318334 time_since_last_state_block=0s
# note: this is the 1.3.0 and earlier format, with a height key
$ ldb --try_load_options --db=/home/dev/.cache/zebra-custom-alt/state/v25/mainnet --secondary_path= --column_family=sprout_note_commitment_tree --hex scan
0x000190 : 0x01030144E5C618CFE9DD46EFF7FAC80C3485C0AD9D8E16D5FF4E50325CFB0D17C15DFD43EF05CFA8A6ADFDE07E274D80284737B900F66D68DC95DE30AE2D7A42E4B9A601780AE637B2A24624C42D79046971DD0157DE93E76CFBF642E57D7D8DBB68BE4D01DF41D2494411D4A7292AFE1DDC0F929AD1817D3D53F099D50AFA9FF755A9EDFD

Upgrade to 1.4.0:

$ cargo install zebrad --version 1.4.0 --locked
$ /home/dev/.cargo/bin/zebrad ~/.config/zebra/zebrad-mainnet-alt.toml
# action: waited until the upgrade was complete - this can take a long time for a large database
2023-11-21T04:58:03.629272Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: trying to open older database format: launching upgrade task running_version=25.3.0 disk_version=25.0.2
2023-11-21T04:58:03.629794Z  INFO zebra_state::service::finalized_state::disk_db: the open file limit is high enough for Zebra current_limit=1024 min_limit=512 ideal_limit=1024
2023-11-21T04:58:03.674534Z  INFO zebra_state::service::finalized_state::disk_db: Opened Zebra state cache at /home/dev/.cache/zebra-custom-alt/state/v25/mainnet
2023-11-21T04:58:03.675398Z  INFO zebra_state::service::finalized_state: loaded Zebra state cache tip=Some((Height(400), block::Hash("0000009761546757b8cfb247829029e4fced863b39d54208a7f4238f23508192")))
2023-11-21T04:58:03.676290Z  INFO zebra_state::service: starting legacy chain check                                                                                                                        
2023-11-21T04:58:03.676413Z  INFO zebra_state::service: cached state consensus branch is valid: no legacy chain found 
...
2023-11-21T04:58:03.679685Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: marked database format as upgraded running_version=25.3.0 disk_version=25.0.2 format_upgrade_version=25.1.1 
2023-11-21T04:58:03.680770Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: marked database format as upgraded running_version=25.3.0 disk_version=25.1.1 format_upgrade_version=25.2.2 
2023-11-21T04:58:03.747159Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: marked database format as upgraded running_version=25.3.0 disk_version=25.2.2 format_upgrade_version=25.3.0
2023-11-21T04:58:03.747167Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: Zebra automatically upgraded the database format to: newer_running_version=25.3.0
2023-11-21T04:58:03.748387Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: database format is valid running_version=25.3.0 inital_disk_version=25.0.2   
# note: this is the 1.4.0 and later format, with an empty key
$ ldb --try_load_options --db=/home/dev/.cache/zebra-custom-alt/state/v25/mainnet --secondary_path= --column_family=sprout_note_commitment_tree --hex scan
0x : 0x01FB712A01675DE9478F49E7B535E2797969B5C536466301F038E9B324FB0B713978B3DB7000562ACB81C32A0676AC80648737B81CD1622AF40FF7B32342EEE8D55BF2D19206DE36EE860B79D5E8644FC4FFF170F797B6C5513D6D9CB7601B50C2E2F92D47D1A5E0908A86B36AB8469055907A0B685DDEB9A77039D2F0E908AB06325CE78D5DBBB511FCCA862FE57315CC96BB29DB7A2C9A10A736B4BC0BCB24597FA7C42590A1EE40A1CD39B77AB96084FFC70E7AB38785014764F8599B9888F38F96168F75AE51B6750E9806C24D7A3585EDCDD60D8C9D5D2BD4DB395FCBD46AF19A95B3404EEA8BE5D18ECA477F59F690D191D9D08ECC9E98D4289E1887917B1BE8F0B4E001EF5A4A23AEBE9642D1B563BD5601180C2D85E595CD1F8BC57439BBB74A1A7447

I wasn't able to reproduce the panic using a 1.1.0 to 1.4.0 upgrade, starting 1.4.0 again also worked.

teor2345 commented 11 months ago

@hhanh00 based on my testing, I think the likely causes are:

  1. Database corruption on disk.
  2. Using unsupported Zebra versions with bugs in this code (including some previous main branch commits).

If you run this command, we might be able to tell if it's database corruption, or something we might be able to fix:

ldb --try_load_options --db=/ext/hanh/.cache/zebra/state/v25/mainnet --secondary_path= --column_family=sprout_note_commitment_tree --hex scan

The install instructions are in this comment: https://github.com/ZcashFoundation/zebra/issues/7961#issuecomment-1820210550

These commands could also help discover database corruption:

ldb --try_load_options --db=/ext/hanh/.cache/zebra/state/v25/mainnet --secondary_path= checkconsistency
hhanh00 commented 11 months ago 
[hanh@archlinux rocksdb]$ ./ldb --try_load_options --db=/ext/hanh/.cache/zebra/state/v25/mainnet --secondary_path --column_family=sprout_note_commitment_tree --hex scan
0x2315A2 ==> 0x01FC1794170001522E660C1B900B44BD6336150869C059EE45159E024DD0593D7D7FF0784A74DB2E8041DE1C3C5F3886839AEB686D75B9969D8FFAA99400D9441AA9A4BBA808450ABF801F6B0BD322933061050FCB07C31CF18E6D7848FFAC4697A0EA4765AC1287AFB3BD108F5FF4331C19426957B6CDEA077FC98D9914A8A77EB608B4448BF67440688A3603EAA68AE016DFE06DA955450E52B4C78DCAD0CD905955400C9F92A229E4B14EB580E8352CCC8100F28B66B5E90FB6807C6D3A5D94A40C13411A538E96295A1E8D018C1C59F098E69350C7A206B850C828710FBAEB86D3EF72DA3BB6806212FF57170F6904D462D24BC5DF6112269A0B65DEFFFBD4B78FF9A7859B9D83A03915C566AB12BA59B130BAB08A25B932CE204BF1774FC9E9267E5130F191465DBB7F98DE7F19E22F37C537660E9447FC27E1839238DC75548B047C41D1BEC14D7B1DFAE9C25C284218D74C43621B795246A5FB3C238C2CA9D7F91D5AA8601B1FC3241793D87854115EC7190EA06C64B76FAA971C77CCC6316EC716F90F7701360304C7831F3035AA62123F8B79C1CDA82934180FF8FEFC8C8030E44B92729C
[hanh@archlinux rocksdb]$ ./ldb --try_load_options --db=/ext/hanh/.cache/zebra/state/v25/mainnet --secondary_path= checkconsistency
OK
teor2345 commented 11 months ago

Analysis for the Zebra team:

The logs say the tip (from hash_by_height) is:

2023-11-21T03:04:22.467992Z INFO zebra_state::service::finalized_state: loaded Zebra state cache tip=Some((Height(2301194), block::Hash("00000000009f26c58b5a8328a22c8d9a691dd70d0e4fc1fef852c6e2edbb109e"))) 2301194

The finalized state sprout tip is:

0x2315A2 ==> ... 2299298

This means that 1896 blocks were committed to the state without updating the sprout tip.

The most likely cause of this bug in an upgrade from 1.1.0 to 1.4.0 is:

  1. The subtree upgrade 25.2.2 runs for 20 minutes
  2. At the same time, Zebra starts syncing and committing a lot of blocks
  3. The tree key type upgrade reads the tip tree for 2299298 here: https://github.com/ZcashFoundation/zebra/blob/5e4c0f973f3a332ae35fb47e5b8678e3732079b2/zebra-state/src/service/finalized_state/disk_format/upgrade/fix_tree_key_type.rs#L25-L26
  4. The state commits 1896 blocks, and writes the tip trees up to 2301194
  5. The cached sprout tip upgrade overwrites the tip tree for 2301194 with the tip tree for 2299298 here (and likely also the history tree): https://github.com/ZcashFoundation/zebra/blob/5e4c0f973f3a332ae35fb47e5b8678e3732079b2/zebra-state/src/service/finalized_state/disk_format/upgrade/fix_tree_key_type.rs#L45-L46
  6. The incorrect trees cause a panic, which leads to a panic-restart loop.

This is most likely to happen in the key type upgrade, because it deletes all the previous keys in the same batch, including reading all the tombstones for all the heights that have already been deleted. So the permanent fix is to move that slow cleanup code into another batch.

Unfortunately it's impossible to avoid this race condition without upgrading the key type, which is what the 25.3.0 upgrade actually does! We could also pause Zebra's sync during the upgrade to avoid the race condition, but that fix isn't needed now, because almost everyone has already upgraded.

So I think we might be able to come up with a manual fix for corrupt databases, let me check with the team.

teor2345 commented 11 months ago

@hhanh00 we also need to check what happened to the other tree that is in the same upgrade. Can you please run this command?

ldb --try_load_options --db=/ext/hanh/.cache/zebra/state/v25/mainnet --secondary_path= --column_family=history_tree --hex scan
hhanh00 commented 11 months ago 
[hanh@archlinux rocksdb]$ ./ldb --try_load_options --db=/ext/hanh/.cache/zebra/state/v25/mainnet --secondary_path= --column_family=history_tree --hex scan
0x ==> 0x00FC8ABD12000CFCFEFF0F0000FEFF0700FDFF0F006FE521EC24F560354969BF10FA9E8BE4877BB9D995355E7A241FF3AF634CF3664D559662895BF1649FB3011CD9AA011C9F600FE22C07FD60D49559791EDCF74ECEF542F800AB0ADCFEDA49684655044C350AAE393E2A0B4DB584EA172A43857A2A7FA846D803E8E6C17D49C879540E45A31298F5EC911D05000000000000000000000000000000000000000000000000FE40BE1900FE3FBE2100FE7AAE0A00AE2935F1DFD8A24AED7C70DF7DE3A668EB7A49B1319880DDE2BBD9031AE5D82F24126405FE513405B2D51C70C59A196FDF3392D1F01F2960AC590FDC766CAB30FEC138090000000000000000000000000000000000FCFDFF110000FDFF1000FCFF1100855BDD5061F550EBC155373D660E0BE5BB93B446AC380E1BBBEBB4E0CC47C6C08E5BF16420C13C65E5A8011CC07A011CFD085034D63308F163BE220E77E6C82EF1531BE152183C2D0AE89FBE5952714E5D534C8DF09724695E9B359B46238D7F4EB4863BD077BE7BF6E1FFDE57C3EE1C880366429E229300000000000000000000000000000000000000000000000000FE40BE2100FE3FBE2200FE4627010024126405FE513405B2D51C70C59A196FDF3392D1F01F2960AC590FDC766CAB30D65E0FC00356654E31CAB5DECE79ED00AAA800D60F7608D274460645DA8BA523FD891E000000000000000000000000000000000000FCFC7F120000FC3F1200FB7F12001167E0CDFF8F255ED4D15568E8760B3E8DEC0A6214241857C838DF6C4169ACA99BC13C65AC984F65EA72011C27C2011CA55BF0F5BB57301B2AE42CF2AA262131485D8589C47BEA82F6CC703510B54317CAE08F3608FA594F7544A7FBD2C2A154AB6AC24DCAC19F4D658EF0D8469491526AE4096FFDE12500000000000000000000000000000000000000000000000000FE40BE2200FE3FFE2200FD5448D65E0FC00356654E31CAB5DECE79ED00AAA800D60F7608D274460645DA8BA523C62E2951EF794E2B80D6ED15167DCA48D8F1CE2DA8B18BD924BC269ADF417800FDC1060000000000000000000000000000000000000000FCFB9F120000FB8F1200FA9F1200412A9083646FBFC6DC3CD3AD7CEC6D41118956E33D815BF4A1EE412B27DA5E8C08994F65664F54655EC4011CC5D7011C57D765DE64AE356E24EAF82B7128DC26FAA1D2080CAF1722DAB79D4EC9F81D6A5DF5BC68B167C91D0019950E568E50AC9DB070CC035977DC3211E027A5D63A3B395F8E361AF00800000000000000000000000000000000000000000000000000FE40FE2200FE3F0E2300FDCE12C62E2951EF794E2B80D6ED15167DCA48D8F1CE2DA8B18BD924BC269ADF417800C892D50EE67AA5DF5402CF21B4FD7EAE96DD18737734A10D3A067656617F363AFDAD010000000000000000000000000000000000000000FCFAAF120000FAA71200F9AF1200B502074C249BE13CF073C1104A2C94A137D338DFF20F83E48AD51048DE200D337C4F54650AAB56653DDC011C2B93011C1542E035254A679FA0C1193015C623BFC93D5F8DB3AFDAD7A1625FB4B1FC091DF25C1BC50A9BAE680D4698752DEF03FEF6E590E5F86293BD981BA3F47996C75EBEDC037E59A70400000000000000000000000000000000000000000000000000FE400E2300FE3F162300FDD605C892D50EE67AA5DF5402CF21B4FD7EAE96DD18737734A10D3A067656617F363A9B9CFCE2A4006C63ADD8CB266F4B5B92E26F9BF79A4A7E0B58AD0CC255DF2232FD47010000000000000000000000000000000000000000FCF9B7120000F9B31200F8B71200B009E82DFEF7BFB09F4263F4CAC9D340984FBAAF7743828498E75AEB8829A1FB7FAB5665E8D45765C18E011C8655011CF25C1BC50A9BAE680D4698752DEF03FEF6E590E5F86293BD981BA3F47996C75E7B0DC3E18F98C8E400003DF67027D00883045F280398F25A8E256E9AF38CAD316FDE02CC7E570200000000000000000000000000000000000000000000000000FE40162300FE3F1A2300FD0B019B9CFCE2A4006C63ADD8CB266F4B5B92E26F9BF79A4A7E0B58AD0CC255DF2232220C0C00E0C5359E427A7314ABA6826B0B21FD255C8912B3F8F3CA7E1665BF097F00000000000000000000000000000000000000000000FCF8BB120000F8B91200F7BB1200B7DA38A909746104B9D2549B09D1A4D9A19B9AA046B513F126B5C630D1768A265DD55765CD715865E458011CC513021C7B0DC3E18F98C8E400003DF67027D00883045F280398F25A8E256E9AF38CAD313A42D21075AD9EE7887A392CA5900225C286866E3A96E17570D49D942170FC3808F308D2B9250100000000000000000000000000000000000000000000000000FE401A2300FE3F1C230065220C0C00E0C5359E427A7314ABA6826B0B21FD255C8912B3F8F3CA7E1665BF0919A47A9EC2CFB566A0B634932F9DAC0A19ADD05F9FB152BE4292CECAE615ED0D33000000000000000000000000000000000000000000000000FCF7BC12000077BC1200F6BC12001A5CC0EBDE535081000A52F00FEB433EE031970A986558D0461D13F695478E871C72586574965865511A021CEDE9011C3A42D21075AD9EE7887A392CA5900225C286866E3A96E17570D49D942170FC38A42FEF8FBE8870C3CF2AB5ED81AE9969AA296FB38AB216F972162BE89F413348631ED9AEFB490000000000000000000000000000000000000000000000000000FE401C2300FEBF1C23001D19A47A9EC2CFB566A0B634932F9DAC0A19ADD05F9FB152BE4292CECAE615ED0D4572505D0B532C8D0738117DA9083C9FA7BB249490B75805933C07803F36EE350A000000000000000000000000000000000000000000000000FC76BD12000036BD120075BD1200586C47EFE3EAA8C82C3F46741C6E18E49FDA27C1F1F0A595B9A6EA4E402DBACD86965865ACA85865BADB011CC8F1011CA42FEF8FBE8870C3CF2AB5ED81AE9969AA296FB38AB216F972162BE89F413348EA23CF465FA7B1EEEB4C8E9D24CDF1E9525661B7EBD01CB1FD340781008C9170EC2D0D4926210000000000000000000000000000000000000000000000000000FEC01C2300FEFF1C2300124572505D0B532C8D0738117DA9083C9FA7BB249490B75805933C07803F36EE35082BBF3AE318ED7F8CD75FA864A2E27FF22B619CCA6EF5E1B5A7489E9F3F6E2304000000000000000000000000000000000000000000000000FC85BD1200007DBD120084BD12003D88E809628DB6139F16A9146F483DAEC1364645FFA2D2C21FD1FC8BE656FB5D0BA9586598AA58655EF1011C29DF011CEA23CF465FA7B1EEEB4C8E9D24CDF1E9525661B7EBD01CB1FD340781008C917010ACB24B9E8B0E9D392E68253CAFC3139C11F294CBCFBE7B6C75E9240CF6342F89085BF73B040000000000000000000000000000000000000000000000000000FE001D2300FE071D230002082BBF3AE318ED7F8CD75FA864A2E27FF22B619CCA6EF5E1B5A7489E9F3F6E23082BBF3AE318ED7F8CD75FA864A2E27FF22B619CCA6EF5E1B5A7489E9F3F6E2300000000000000000000000000000000000000000000000000FC88BD12000086BD120087BD12002534A1210B333617620EB3B1CA4B1E6E67DC538093C5C5228C89671699141D1A9EAA5865C5AA58650DE5011C0CDC011C10ACB24B9E8B0E9D392E68253CAFC3139C11F294CBCFBE7B6C75E9240CF6342F10ACB24B9E8B0E9D392E68253CAFC3139C11F294CBCFBE7B6C75E9240CF6342FB99E5BC710010000000000000000000000000000000000000000000000000000FE081D2300FE091D230000082BBF3AE318ED7F8CD75FA864A2E27FF22B619CCA6EF5E1B5A7489E9F3F6E23082BBF3AE318ED7F8CD75FA864A2E27FF22B619CCA6EF5E1B5A7489E9F3F6E2300000000000000000000000000000000000000000000000000FC89BD1200019E10BBEDE2C652F8FEC14F0E0DD71D699A8D2CA228835A8BC5269F000000000029AB586529AB586525E0011C25E0011C7F37EAAAE0AB87E10EAC26A8BA5DCE095CB05AD4EDC89F7252C79FBCD57B9B667F37EAAAE0AB87E10EAC26A8BA5DCE095CB05AD4EDC89F7252C79FBCD57B9B662916037E88000000000000000000000000000000000000000000000000000000FE0A1D2300FE0A1D230001082BBF3AE318ED7F8CD75FA864A2E27FF22B619CCA6EF5E1B5A7489E9F3F6E23082BBF3AE318ED7F8CD75FA864A2E27FF22B619CCA6EF5E1B5A7489E9F3F6E23000000000000000000000000000000000000000000000000000000000000000000FC0A1D2300
0x2315A2 ==> 0x00FCBBAE12000BFCFEFF0F0000FEFF0700FDFF0F006FE521EC24F560354969BF10FA9E8BE4877BB9D995355E7A241FF3AF634CF3664D559662895BF1649FB3011CD9AA011C9F600FE22C07FD60D49559791EDCF74ECEF542F800AB0ADCFEDA49684655044C350AAE393E2A0B4DB584EA172A43857A2A7FA846D803E8E6C17D49C879540E45A31298F5EC911D05000000000000000000000000000000000000000000000000FE40BE1900FE3FBE2100FE7AAE0A00AE2935F1DFD8A24AED7C70DF7DE3A668EB7A49B1319880DDE2BBD9031AE5D82F24126405FE513405B2D51C70C59A196FDF3392D1F01F2960AC590FDC766CAB30FEC138090000000000000000000000000000000000FCFDFF110000FDFF1000FCFF1100855BDD5061F550EBC155373D660E0BE5BB93B446AC380E1BBBEBB4E0CC47C6C08E5BF16420C13C65E5A8011CC07A011CFD085034D63308F163BE220E77E6C82EF1531BE152183C2D0AE89FBE5952714E5D534C8DF09724695E9B359B46238D7F4EB4863BD077BE7BF6E1FFDE57C3EE1C880366429E229300000000000000000000000000000000000000000000000000FE40BE2100FE3FBE2200FE4627010024126405FE513405B2D51C70C59A196FDF3392D1F01F2960AC590FDC766CAB30D65E0FC00356654E31CAB5DECE79ED00AAA800D60F7608D274460645DA8BA523FD891E000000000000000000000000000000000000FCFC7F120000FC3F1200FB7F12001167E0CDFF8F255ED4D15568E8760B3E8DEC0A6214241857C838DF6C4169ACA99BC13C65AC984F65EA72011C27C2011CA55BF0F5BB57301B2AE42CF2AA262131485D8589C47BEA82F6CC703510B54317CAE08F3608FA594F7544A7FBD2C2A154AB6AC24DCAC19F4D658EF0D8469491526AE4096FFDE12500000000000000000000000000000000000000000000000000FE40BE2200FE3FFE2200FD5448D65E0FC00356654E31CAB5DECE79ED00AAA800D60F7608D274460645DA8BA523C62E2951EF794E2B80D6ED15167DCA48D8F1CE2DA8B18BD924BC269ADF417800FDC1060000000000000000000000000000000000000000FCFB9F120000FB8F1200FA9F1200412A9083646FBFC6DC3CD3AD7CEC6D41118956E33D815BF4A1EE412B27DA5E8C08994F65664F54655EC4011CC5D7011C57D765DE64AE356E24EAF82B7128DC26FAA1D2080CAF1722DAB79D4EC9F81D6A5DF5BC68B167C91D0019950E568E50AC9DB070CC035977DC3211E027A5D63A3B395F8E361AF00800000000000000000000000000000000000000000000000000FE40FE2200FE3F0E2300FDCE12C62E2951EF794E2B80D6ED15167DCA48D8F1CE2DA8B18BD924BC269ADF417800C892D50EE67AA5DF5402CF21B4FD7EAE96DD18737734A10D3A067656617F363AFDAD010000000000000000000000000000000000000000FCFAA7120000FAA31200F9A71200FF05FCA8DFA0783762E82BEDC2527FC73071D9C63BDDF1BBE12A71B5615BA7DD7C4F54654E7E55653DDC011C60FB011C1542E035254A679FA0C1193015C623BFC93D5F8DB3AFDAD7A1625FB4B1FC091D2B5B6B75E5632A7314FC4506BE10D8B53228F8DD84F796AEC429B48E74CC0D17929D62F8BE520200000000000000000000000000000000000000000000000000FE400E2300FE3F122300FD6104C892D50EE67AA5DF5402CF21B4FD7EAE96DD18737734A10D3A067656617F363A47CFEB9C97E7E055965E979C1FF744ECC20235952A17E3CA241DC1F7FAB9D92ED100000000000000000000000000000000000000000000FCF9AB120000F9A91200F8AB1200C9D6551919563B2FE9BC7C4A3A7B1971B84A1D2EAEFE31F6C8D3314DA79EB110007F55659B15566562F0011C71E9011C8487E51E4A0D747A717926086E755193F29627C9A829215389055162C402485427CF0FCF3D51E84B7376D21F370F12A848ED9BF52D04C8166AB0B6493A486A1351B9CB0033320100000000000000000000000000000000000000000000000000FE40122300FE3F142300D447CFEB9C97E7E055965E979C1FF744ECC20235952A17E3CA241DC1F7FAB9D92EFC7D6A7684B2A723D7F1C9B4A44339200DB22DFA675683581466E548DAB5AE0138000000000000000000000000000000000000000000000000FCF8AD120000F8AC1200F7AD12000BDA4133BDC81916FC4E81B78AF73C33EC0A660DE7037CA7F02C652EFDA0A602F11556651D605665ABED011C1BE5011C27CF0FCF3D51E84B7376D21F370F12A848ED9BF52D04C8166AB0B6493A486A137994225020E9552018887613C1A08255E5BC123DB67C24C8F4604E449F33CA5E130107ECB3900000000000000000000000000000000000000000000000000000FE40142300FE3F1523005A877F46D786F7F201F8058CDA07CE5C3AF3B5D69E122C91A777B6A62B32CC3E0AC6199D31B71CC03E2853A6BFED8F80247C1193F21161806346FFC27F13011F0621000000000000000000000000000000000000000000000000FC77AE12000037AE120076AE120030FAAB0B76FA49CD67F9583AECC19C3157229683475A9BBF460881383D7CCEBC6D605665FD72566555E1011C38D6011C7994225020E9552018887613C1A08255E5BC123DB67C24C8F4604E449F33CA5E7F9C0682E93CF4358918ABF5F5697C230633429F32F81EAB3460E73364AF921F8C771B2A53220000000000000000000000000000000000000000000000000000FE40152300FE7F15230012AE10F348F0C2D4ED7D086A5C7AA789B2E46644D6AEAE1D013EE24E12D35E290B434A69380F75FABBF03845D6FCB21EA6DFD27BB2A687A4F4E38EE008D643282706000000000000000000000000000000000000000000000000FCB6AE12000096AE1200B5AE1200A33CACBB798C64745DC4FCC4F65194BD84DAAC48CE65997638CD9F7B96F92C65347356651A7B5665F3D5011C68B1011C7F9C0682E93CF4358918ABF5F5697C230633429F32F81EAB3460E73364AF921FC7AC5608211670B69AC6802AA17031E27EBE74092A3EB3C8D9E8E7C59BB91D5F4E89631B74120000000000000000000000000000000000000000000000000000FE80152300FE9F1523000A434A69380F75FABBF03845D6FCB21EA6DFD27BB2A687A4F4E38EE008D643282723AF91AA5C9167428606C956B3AD30EC3FB6A19C4E731A61D60B95C16E7D4A2103000000000000000000000000000000000000000000000000FCB9AE120000B7AE1200B8AE12004F32755634B2E05510CBA55E461AF34B68B2A4914CE7F5B488A8813B21D7AF048E7B5665927B566519BD011C69B8011CC7AC5608211670B69AC6802AA17031E27EBE74092A3EB3C8D9E8E7C59BB91D5FC7AC5608211670B69AC6802AA17031E27EBE74092A3EB3C8D9E8E7C59BB91D5F30DEDC0B28010000000000000000000000000000000000000000000000000000FEA0152300FEA11523000023AF91AA5C9167428606C956B3AD30EC3FB6A19C4E731A61D60B95C16E7D4A2123AF91AA5C9167428606C956B3AD30EC3FB6A19C4E731A61D60B95C16E7D4A2100000000000000000000000000000000000000000000000000FCBAAE12000156A44E2B8821A1E09CBA74318FD245DC9D8A53A98169488429C5140000000000BC7B5665BC7B566504B5011C04B5011CC7AC5608211670B69AC6802AA17031E27EBE74092A3EB3C8D9E8E7C59BB91D5FC7AC5608211670B69AC6802AA17031E27EBE74092A3EB3C8D9E8E7C59BB91D5F790C6DF695000000000000000000000000000000000000000000000000000000FEA2152300FEA215230000037DA077C80ACE206233E9821319E4B7374A149676DF77A4B7339E79887E0A26037DA077C80ACE206233E9821319E4B7374A149676DF77A4B7339E79887E0A26010000000000000000000000000000000000000000000000000000000000000000FCA2152300
teor2345 commented 11 months ago

Ok, yep, it seems like the latest history tree has been written, but there's also a history tree for height 2299298 that wasn't cleaned up. That should be sorted when the 25.3.0 upgrade runs.

For the Zebra team:

I think the bug is much simpler:

We only write the sprout tree when there is a sprout transaction: https://github.com/ZcashFoundation/zebra/blob/5e4c0f973f3a332ae35fb47e5b8678e3732079b2/zebra-state/src/service/finalized_state/zebra_db/shielded.rs#L520-L523

But we expect the sprout tree to always be at the tip height: https://github.com/ZcashFoundation/zebra/blob/5e4c0f973f3a332ae35fb47e5b8678e3732079b2/zebra-state/src/service/finalized_state/zebra_db/shielded.rs#L111-L117

This isn't true since upgrade 1.4.0, so the fix is to find the tree with the greatest height, just like we do for Sapling and Orchard.

teor2345 commented 11 months ago

@hhanh00 does the fix in PR #7972 fix this bug for you?

@upbqdn has reproduced the bug on testnet and that fix worked for him.

hhanh00 commented 11 months ago

It doesn't build with rust stable.

error[E0658]: use of unstable library feature 'int_roundings'
   --> zebra-network/src/address_book.rs:276:41
    |
276 |         let address_limit = peers.len().div_ceil(ADDR_RESPONSE_LIMIT_DENOMINATOR);
    |                                         ^^^^^^^^
    |
    = note: see issue #88581 <https://github.com/rust-lang/rust/issues/88581> for more information
teor2345 commented 11 months ago

It doesn't build with rust stable.

div_ceil was stabilised in Rust 1.73, which was released 7 weeks ago: https://releases.rs/docs/1.73.0/#stabilized-apis

The current stable is Rust 1.74: https://releases.rs/

We can replace that code if you're unable to upgrade your Rust version.

hhanh00 commented 11 months ago

All good, thanks!

teor2345 commented 11 months ago

Thanks, but we usually close tickets when the PR merges and the bug is fixed for everyone.