search

nautechsystems / nautilus_trader

A high-performance algorithmic trading platform and event-driven backtester
https://nautilustrader.io
GNU Lesser General Public License v3.0
1.71k stars 402 forks source link

Timer is panicking when calling `set_time_alert_ns` #1640

Closed davidsblom closed 1 month ago

davidsblom commented 1 month ago

Bug Report

During a sandbox paper trading test, the timer panics when calling set_time_alert_ns. The timer is called by a new throttler which is created by the strategy directly.

The throttler is used to limit the number of limit orders sent to the exchange. Bybit has a limit of 10 orders per second. The strategy is creating orders on_quote_tick which sometimes creates too many orders.

The error happens immediately after when the throttler drops a message. Then, it tries to set a new timer I believe.

Expected Behavior

No panics. It is probably already useful if an exception is thrown in Python or Cython. Rust is complaining that 'interval_ns' is not positive, was 0 (haven't figured out yet how that is even possible :)

Actual Behavior

thread '<unnamed>' panicked at common/src/ffi/clock.rs:362:10:
called `Result::unwrap()` on an `Err` value: Condition failed: invalid u64 for 'interval_ns' not positive, was 0

Stack backtrace:
   0: std::backtrace_rs::backtrace::libunwind::trace
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1: std::backtrace_rs::backtrace::trace_unsynchronized
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2: std::backtrace::Backtrace::create
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/backtrace.rs:331:13
   3: anyhow::error::<impl anyhow::Error>::msg
             at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.83/src/error.rs:83:36
   4: anyhow::__private::format_err
             at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.83/src/lib.rs:691:13
   5: nautilus_core::correctness::check_positive_u64
             at ./nautilus_trader/nautilus_core/core/src/correctness.rs:116:9
   6: nautilus_common::timer::LiveTimer::new
             at ./nautilus_trader/nautilus_core/common/src/timer.rs:253:9
   7: <nautilus_common::clock::LiveClock as nautilus_common::clock::Clock>::set_time_alert_ns
             at ./nautilus_trader/nautilus_core/common/src/clock.rs:340:25
   8: live_clock_set_time_alert
             at ./nautilus_trader/nautilus_core/common/src/ffi/clock.rs:360:5
   9: __pyx_f_15nautilus_trader_6common_9component_9LiveClock_set_time_alert_ns
             at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:40520:3
  10: __pyx_f_15nautilus_trader_6common_9component_9Throttler__set_timer
             at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:73668:3
  11: __pyx_f_15nautilus_trader_6common_9component_9Throttler__limit_msg
             at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:73406:5
  12: __pyx_f_15nautilus_trader_6common_9component_9Throttler_send
             at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:72867:5
  13: __pyx_pf_15nautilus_trader_6common_9component_9Throttler_6send
             at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:72998:3
  14: __pyx_pw_15nautilus_trader_6common_9component_9Throttler_7send
             at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:72976:13
  15: _PyObject_Vectorcall
  16: __PyEval_EvalFrameDefault
  17: __PyEval_Vector
  18: __pyx_f_15nautilus_trader_6common_5actor_5Actor_on_quote_tick
             at ./nautilus_trader/build/optimized/nautilus_trader/common/actor.c:33156:23
  19: __pyx_f_15nautilus_trader_6common_5actor_5Actor_handle_quote_tick
             at ./nautilus_trader/build/optimized/nautilus_trader/common/actor.c:62480:9
  20: __pyx_pf_15nautilus_trader_6common_5actor_5Actor_174handle_quote_tick
             at ./nautilus_trader/build/optimized/nautilus_trader/common/actor.c:62763:3
  21: __pyx_pw_15nautilus_trader_6common_5actor_5Actor_175handle_quote_tick
             at ./nautilus_trader/build/optimized/nautilus_trader/common/actor.c:62737:13
  22: __pyx_f_15nautilus_trader_6common_9component_10MessageBus_publish_c
             at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:67425:19
  23: __pyx_f_15nautilus_trader_4data_6engine_10DataEngine__handle_quote_tick
             at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58659:3
  24: __pyx_pf_15nautilus_trader_4data_6engine_10DataEngine_122_handle_quote_tick
             at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58794:3
  25: __pyx_pw_15nautilus_trader_4data_6engine_10DataEngine_123_handle_quote_tick
             at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58768:13
  26: __pyx_f_15nautilus_trader_4data_6engine_10DataEngine__handle_quote_tick
             at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58467:23
  27: __pyx_f_15nautilus_trader_4data_6engine_10DataEngine__handle_data
             at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:56583:5
  28: __pyx_pf_15nautilus_trader_4data_6engine_10DataEngine_112_handle_data
             at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:57015:3
  29: __pyx_pw_15nautilus_trader_4data_6engine_10DataEngine_113_handle_data
             at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:56989:13
  30: _PyObject_Vectorcall
  31: __PyEval_EvalFrameDefault
  32: _gen_send_ex2
  33: _task_step
  34: _task_wakeup
  35: _cfunction_vectorcall_O
  36: ___pyx_f_6uvloop_4loop_6Handle__run
  37: ___pyx_f_6uvloop_4loop_4Loop__on_idle
  38: ___pyx_f_6uvloop_4loop_6Handle__run
  39: ___pyx_f_6uvloop_4loop_cb_idle_callback
  40: _uv__run_idle
  41: _uv_run
  42: ___pyx_f_6uvloop_4loop_4Loop___run
  43: ___pyx_f_6uvloop_4loop_4Loop__run
  44: ___pyx_pw_6uvloop_4loop_4Loop_25run_forever
  45: ___Pyx_PyObject_CallNoArg
  46: ___pyx_pw_6uvloop_4loop_4Loop_45run_until_complete
  47: _method_vectorcall_O
  48: _PyObject_Vectorcall
  49: __PyEval_EvalFrameDefault
  50: _PyEval_EvalCode
  51: _run_mod
  52: __PyRun_SimpleFileObject
  53: __PyRun_AnyFileObject
  54: _Py_RunMain
  55: _pymain_main
  56: _Py_BytesMain
stack backtrace:
   0:        0x30576ca6c - std::backtrace_rs::backtrace::libunwind::trace::hadefa2fe489b32be
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:        0x30576ca6c - std::backtrace_rs::backtrace::trace_unsynchronized::h37e8bb3c3fd9c67c
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x30576ca6c - std::sys_common::backtrace::_print_fmt::h4994a90b70e9aaa7
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:68:5
   3:        0x30576ca6c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb478ebbfb46e27ce
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:44:22
   4:        0x3057aba74 - core::fmt::rt::Argument::fmt::hfe2e626e52db21d5
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/rt.rs:142:9
   5:        0x3057aba74 - core::fmt::write::he4d5fa2daff1f531
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/mod.rs:1120:17
   6:        0x305763274 - std::io::Write::write_fmt::hc5a47a68eba63d9f
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/io/mod.rs:1810:15
   7:        0x30576c8a0 - std::sys_common::backtrace::_print::hd47355bf70e1c2da
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:47:5
   8:        0x30576c8a0 - std::sys_common::backtrace::print::h79bd952cc5812e7a
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:34:9
   9:        0x30576f368 - std::panicking::default_hook::{{closure}}::h82301f6222887737
  10:        0x30576f0b0 - std::panicking::default_hook::h1e49abbb3f1d7dbf
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:292:9
  11:        0x30576f8c8 - std::panicking::rust_panic_with_hook::h1e70c5d905e30e9d
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:779:13
  12:        0x30576f7bc - std::panicking::begin_panic_handler::{{closure}}::h399e32952efd26a4
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
  13:        0x30576cef0 - std::sys_common::backtrace::__rust_end_short_backtrace::h2ab87f841a2323e7
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
  14:        0x30576f544 - rust_begin_unwind
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
  15:        0x305817acc - core::panicking::panic_fmt::h33e40d2a93cab78f
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
  16:        0x305817f0c - core::result::unwrap_failed::h79c2cb9e4fc28c72
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/result.rs:1649:5
  17:        0x3051ee5b0 - core::result::Result<T,E>::unwrap::h81d0ea2beeb4b9b4
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/result.rs:1073:23
  18:        0x30496eb4c - live_clock_set_time_alert
                               at /Users/davidblom/repositories/market-master/nautilus_trader/nautilus_core/common/src/ffi/clock.rs:360:5
  19:        0x304855fd8 - __pyx_f_15nautilus_trader_6common_9component_9LiveClock_set_time_alert_ns
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:40520:3
  20:        0x30486e6ac - __pyx_f_15nautilus_trader_6common_9component_9Throttler__set_timer
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:73668:3
  21:        0x30486de74 - __pyx_f_15nautilus_trader_6common_9component_9Throttler__limit_msg
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:73406:5
  22:        0x30486d534 - __pyx_f_15nautilus_trader_6common_9component_9Throttler_send
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:72867:5
  23:        0x30489347c - __pyx_pf_15nautilus_trader_6common_9component_9Throttler_6send
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:72998:3
  24:        0x30489347c - __pyx_pw_15nautilus_trader_6common_9component_9Throttler_7send
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:72976:13
  25:        0x1032998ac - _PyObject_Vectorcall
  26:        0x103379d38 - __PyEval_EvalFrameDefault
  27:        0x103371340 - __PyEval_Vector
  28:        0x167b15b00 - __pyx_f_15nautilus_trader_6common_5actor_5Actor_on_quote_tick
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/actor.c:33156:23
  29:        0x167b3bb24 - __pyx_f_15nautilus_trader_6common_5actor_5Actor_handle_quote_tick
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/actor.c:62480:9
  30:        0x167b5d24c - __pyx_pf_15nautilus_trader_6common_5actor_5Actor_174handle_quote_tick
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/actor.c:62763:3
  31:        0x167b5d24c - __pyx_pw_15nautilus_trader_6common_5actor_5Actor_175handle_quote_tick
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/actor.c:62737:13
  32:        0x30486afb0 - __pyx_f_15nautilus_trader_6common_9component_10MessageBus_publish_c
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:67425:19
  33:        0x17e4e8508 - __pyx_f_15nautilus_trader_4data_6engine_10DataEngine__handle_quote_tick
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58659:3
  34:        0x17e5070ac - __pyx_pf_15nautilus_trader_4data_6engine_10DataEngine_122_handle_quote_tick
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58794:3
  35:        0x17e5070ac - __pyx_pw_15nautilus_trader_4data_6engine_10DataEngine_123_handle_quote_tick
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58768:13
  36:        0x17e4e8784 - __pyx_f_15nautilus_trader_4data_6engine_10DataEngine__handle_quote_tick
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58467:23
  37:        0x17e4e5424 - __pyx_f_15nautilus_trader_4data_6engine_10DataEngine__handle_data
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:56583:5
  38:        0x17e5064cc - __pyx_pf_15nautilus_trader_4data_6engine_10DataEngine_112_handle_data
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:57015:3
  39:        0x17e5064cc - __pyx_pw_15nautilus_trader_4data_6engine_10DataEngine_113_handle_data
                               at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:56989:13
  40:        0x1032998ac - _PyObject_Vectorcall
  41:        0x103379d38 - __PyEval_EvalFrameDefault
  42:        0x1032b27fc - _gen_send_ex2
  43:        0x103048dbc - _task_step
  44:        0x10304a15c - _task_wakeup
  45:        0x1032e4184 - _cfunction_vectorcall_O
  46:        0x17f881d3c - ___pyx_f_6uvloop_4loop_6Handle__run
  47:        0x17f86d810 - ___pyx_f_6uvloop_4loop_4Loop__on_idle
  48:        0x17f881ff4 - ___pyx_f_6uvloop_4loop_6Handle__run
  49:        0x17f8b343c - ___pyx_f_6uvloop_4loop_cb_idle_callback
  50:        0x17f932288 - _uv__run_idle
  51:        0x17f92c770 - _uv_run
  52:        0x17f86e1e0 - ___pyx_f_6uvloop_4loop_4Loop___run
  53:        0x17f86e474 - ___pyx_f_6uvloop_4loop_4Loop__run
  54:        0x17f8bd850 - ___pyx_pw_6uvloop_4loop_4Loop_25run_forever
  55:        0x17f862098 - ___Pyx_PyObject_CallNoArg
  56:        0x17f8c01bc - ___pyx_pw_6uvloop_4loop_4Loop_45run_until_complete
  57:        0x1032a5128 - _method_vectorcall_O
  58:        0x1032998ac - _PyObject_Vectorcall
  59:        0x103379d38 - __PyEval_EvalFrameDefault
  60:        0x103371210 - _PyEval_EvalCode
  61:        0x1033c40b8 - _run_mod
  62:        0x1033c25ac - __PyRun_SimpleFileObject
  63:        0x1033c2034 - __PyRun_AnyFileObject
  64:        0x1033e1450 - _Py_RunMain
  65:        0x1033e1800 - _pymain_main
  66:        0x1033e18a0 - _Py_BytesMain
fatal runtime error: failed to initiate panic, error 5

Steps to Reproduce the Problem

  1. Sandbox paper trading
  2. Send limit orders on every quote tick
  3. Throttle the limit orders with a throttler to not send too many orders.

Specifications

davidsblom commented 1 month ago

My hunch is that the _delta_next method of the throttler returns a zero value or very small number. Then those microseconds have already passed by the time the Rust timer starts.

davidsblom commented 1 month ago

Maybe it is fine if the timer allows a small negative or zero interval? If that happens, then immediately call the callback without using a thread and mark the timer as expired? The exception should still be thrown if the interval is too negative.

Not sure if this makes sense at all. The alternative would be to use a minimum time interval in the throttler which would introduce a sometimes unneeded delay.

cjdsellers commented 1 month ago

Thanks for all the info @davidsblom

This one shouldn't be too difficult.

davidsblom commented 1 month ago

Done some more testing, and sometimes also stop_time was < now_ns happens.

thread 'tokio-runtime-worker' panicked at thread 'tokio-runtime-worker' panicked at common/src/timer.rs:315:17:
stop_time was < now_ns
stack backtrace:
common/src/timer.rs:315:17:
stop_time was < now_ns
   0:        0x155becaec - std::backtrace_rs::backtrace::libunwind::trace::hadefa2fe489b32be
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:        0x155becaec - std::backtrace_rs::backtrace::trace_unsynchronized::h37e8bb3c3fd9c67c
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x155becaec - std::sys_common::backtrace::_print_fmt::h4994a90b70e9aaa7
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:68:5
   3:        0x155becaec - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb478ebbfb46e27ce
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:44:22
   4:        0x155c2baf4 - core::fmt::rt::Argument::fmt::hfe2e626e52db21d5
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/rt.rs:142:9
   5:        0x155c2baf4 - core::fmt::write::he4d5fa2daff1f531
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/mod.rs:1120:17
   6:        0x155be32f4 - std::io::Write::write_fmt::hc5a47a68eba63d9f
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/io/mod.rs:1810:15
   7:        0x155bec920 - std::sys_common::backtrace::_print::hd47355bf70e1c2da
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:47:5
   8:        0x155bec920 - std::sys_common::backtrace::print::h79bd952cc5812e7a
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:34:9
   9:        0x155bef3e8 - std::panicking::default_hook::{{closure}}::h82301f6222887737
  10:        0x155bef130 - std::panicking::default_hook::h1e49abbb3f1d7dbf
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:292:9
  11:        0x155bef948 - std::panicking::rust_panic_with_hook::h1e70c5d905e30e9d
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:779:13
  12:        0x155bef810 - std::panicking::begin_panic_handler::{{closure}}::h399e32952efd26a4
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:649:13
  13:        0x155becf70 - std::sys_common::backtrace::__rust_end_short_backtrace::h2ab87f841a2323e7
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
  14:        0x155bef5c4 - rust_begin_unwind
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
  15:        0x155c97b4c - core::panicking::panic_fmt::h33e40d2a93cab78f
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
  16:        0x154e0cbdc - nautilus_common::timer::LiveTimer::start::{{closure}}::habba92d2a4238457
                               at /Users/davidblom/repositories/market-master/nautilus_trader/nautilus_core/common/src/timer.rs:315:17
  17:        0x154d48b2c - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h0d2fa539e1e1b7f1
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  18:        0x154d48798 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h561cb9fb250495a4
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  19:        0x154d48798 - tokio::runtime::task::core::Core<T,S>::poll::h2c62a4c97c7bbdec
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
  20:        0x154d80aa8 - tokio::runtime::task::harness::poll_future::{{closure}}::h0436336731ad585f
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  21:        0x154d9aeb8 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hbfb5aaa8c7b25d79
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
  22:        0x154dd4634 - std::panicking::try::do_call::h1446aee4bafdd9ec
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
  23:        0x154dd5010 - ___rust_try
  24:        0x154dd4438 - std::panicking::try::hc607503b50ecf132
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  25:        0x154d921a8 - std::panic::catch_unwind::hb8c43dbb92c3ceef
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  26:        0x154d808e8 - tokio::runtime::task::harness::poll_future::h71b9ed50b8b47066
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  27:        0x154d80d5c - tokio::runtime::task::harness::Harness<T,S>::poll_inner::ha1ce53a7c1f58773
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  28:        0x154d81440 - tokio::runtime::task::harness::Harness<T,S>::poll::h2da67552fda7edb9
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  29:        0x154dcbfa0 - tokio::runtime::task::raw::poll::h2f6358036e21e2cf
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
  30:        0x154eb04b0 - tokio::runtime::task::raw::RawTask::poll::hafbff39549565f0b
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  31:        0x154eb944c - tokio::runtime::task::LocalNotified<S>::run::h246bca158db8637b
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
  32:        0x154eabfbc - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h6070f5802fb1ec23
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:576:13
  33:        0x154eabea0 - tokio::runtime::coop::with_budget::h7fc1cf45bd777a8a
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
  34:        0x154eabea0 - tokio::runtime::coop::budget::h2d07096d46097b77
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
  35:        0x154eabea0 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h28474f49af5b4a1a
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
  36:        0x154eab7ac - tokio::runtime::scheduler::multi_thread::worker::Context::run::h4b51da22208e01f3
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
  37:        0x154eab53c - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::hef4f7dbccbd300b4
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
  38:        0x154f16248 - tokio::runtime::context::scoped::Scoped<T>::set::h48967c3e1e3c9416
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
  39:        0x154ef7318 - tokio::runtime::context::set_scheduler::{{closure}}::ha85412cf3c5d248c
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26
  40:        0x154e989c8 - std::thread::local::LocalKey<T>::try_with::hf6966b26ac81f482
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/local.rs:270:16
  41:        0x154e95144 - std::thread::local::LocalKey<T>::with::hcf8f7976ac6e8c37
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/local.rs:246:9
  42:        0x154ef725c - tokio::runtime::context::set_scheduler::h4e98e3ae7b2ac870
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:9
  43:        0x154eab460 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h9ef275b241e76932
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
  44:        0x154f1aa3c - tokio::runtime::context::runtime::enter_runtime::h2dad1aac06750521
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
  45:        0x154eab2a0 - tokio::runtime::scheduler::multi_thread::worker::run::h0785b03457e9312c
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
  46:        0x154eab14c - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::hfc6e621eac9a0a4b
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
  47:        0x154ee4eec - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h9de3dca6c6d00643
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
  48:        0x154e5d940 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hfec594de548332cb
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  49:        0x154e5b7ac - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hb93a0dbdba224e7a
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  50:        0x154e5b7ac - tokio::runtime::task::core::Core<T,S>::poll::hd25500e57906fe49
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
  51:        0x154e50a60 - tokio::runtime::task::harness::poll_future::{{closure}}::h96d98bbba712eade
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  52:        0x154edb944 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h531849d3adde8f5f
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
  53:        0x154e36c04 - std::panicking::try::do_call::h1ec80eed4e700941
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
  54:        0x154e3db94 - ___rust_try
  55:        0x154e3102c - std::panicking::try::h08e79c44f7072e87
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  56:        0x154efff28 - std::panic::catch_unwind::h35fa19fb2c869dfb
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  57:        0x154e4d884 - tokio::runtime::task::harness::poll_future::h739e178f11ff8fd0
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  58:        0x154e417a8 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h3721f6c38aeae65d
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  59:        0x154e409c4 - tokio::runtime::task::harness::Harness<T,S>::poll::hb8fa6cc544bf5888
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  60:        0x154eb0b00 - tokio::runtime::task::raw::poll::h82f7e00145ca6939
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
  61:        0x154eb04b0 - tokio::runtime::task::raw::RawTask::poll::hafbff39549565f0b
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  62:        0x154eb9548 - tokio::runtime::task::UnownedTask<S>::run::h325b97c52029e9a1
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
  63:        0x154e8c928 - tokio::runtime::blocking::pool::Task::run::haf89b44256125981
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
  64:        0x154e94000 - tokio::runtime::blocking::pool::Inner::run::h861cf3bdaec96f81
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
  65:        0x154e93db4 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h4d4597093fed2b6e
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13
  66:        0x154f25184 - std::sys_common::backtrace::__rust_begin_short_backtrace::h5e5ad456e93498fa
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:155:18
  67:        0x154edae78 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h023866687c51ede6
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:529:17
  68:        0x154edb814 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h46685ce984f28085
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
  69:        0x154e3959c - std::panicking::try::do_call::hf5618dba7a87df90
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
  70:        0x154e3db94 - ___rust_try
  71:        0x154e343a0 - std::panicking::try::h9a5be4bdbb5cc6aa
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  72:        0x154edacfc - std::panic::catch_unwind::hbd10baa2f7d47eec
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  73:        0x154edacfc - std::thread::Builder::spawn_unchecked_::{{closure}}::hf82a72b81ac2d572
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:528:30
  74:        0x154e6d0a8 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h30f9d5df81b8dbf0
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:250:5
  75:        0x155bf7d28 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h122931d29f919766
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  76:        0x155bf7d28 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h222f385c86e71c13
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  77:        0x155bf7d28 - std::sys::unix::thread::Thread::new::thread_start::h8dda2af345bab446
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  78:        0x181b8af94 - __pthread_joiner_wake
stack backtrace:
   0:        0x155becaec - std::backtrace_rs::backtrace::libunwind::trace::hadefa2fe489b32be
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:        0x155becaec - std::backtrace_rs::backtrace::trace_unsynchronized::h37e8bb3c3fd9c67c
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x155becaec - std::sys_common::backtrace::_print_fmt::h4994a90b70e9aaa7
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:68:5
   3:        0x155becaec - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb478ebbfb46e27ce
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:44:22
   4:        0x155c2baf4 - core::fmt::rt::Argument::fmt::hfe2e626e52db21d5
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/rt.rs:142:9
   5:        0x155c2baf4 - core::fmt::write::he4d5fa2daff1f531
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/mod.rs:1120:17
   6:        0x155be32f4 - std::io::Write::write_fmt::hc5a47a68eba63d9f
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/io/mod.rs:1810:15
   7:        0x155bec920 - std::sys_common::backtrace::_print::hd47355bf70e1c2da
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:47:5
   8:        0x155bec920 - std::sys_common::backtrace::print::h79bd952cc5812e7a
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:34:9
   9:        0x155bef3e8 - std::panicking::default_hook::{{closure}}::h82301f6222887737
  10:        0x155bef130 - std::panicking::default_hook::h1e49abbb3f1d7dbf
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:292:9
  11:        0x155bef948 - std::panicking::rust_panic_with_hook::h1e70c5d905e30e9d
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:779:13
  12:        0x155bef810 - std::panicking::begin_panic_handler::{{closure}}::h399e32952efd26a4
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:649:13
  13:        0x155becf70 - std::sys_common::backtrace::__rust_end_short_backtrace::h2ab87f841a2323e7
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
  14:        0x155bef5c4 - rust_begin_unwind
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
  15:        0x155c97b4c - core::panicking::panic_fmt::h33e40d2a93cab78f
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
  16:        0x154e0cbdc - nautilus_common::timer::LiveTimer::start::{{closure}}::habba92d2a4238457
                               at /Users/davidblom/repositories/market-master/nautilus_trader/nautilus_core/common/src/timer.rs:315:17
  17:        0x154d48b2c - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h0d2fa539e1e1b7f1
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  18:        0x154d48798 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h561cb9fb250495a4
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  19:        0x154d48798 - tokio::runtime::task::core::Core<T,S>::poll::h2c62a4c97c7bbdec
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
  20:        0x154d80aa8 - tokio::runtime::task::harness::poll_future::{{closure}}::h0436336731ad585f
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  21:        0x154d9aeb8 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hbfb5aaa8c7b25d79
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
  22:        0x154dd4634 - std::panicking::try::do_call::h1446aee4bafdd9ec
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
  23:        0x154dd5010 - ___rust_try
  24:        0x154dd4438 - std::panicking::try::hc607503b50ecf132
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  25:        0x154d921a8 - std::panic::catch_unwind::hb8c43dbb92c3ceef
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  26:        0x154d808e8 - tokio::runtime::task::harness::poll_future::h71b9ed50b8b47066
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  27:        0x154d80d5c - tokio::runtime::task::harness::Harness<T,S>::poll_inner::ha1ce53a7c1f58773
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  28:        0x154d81440 - tokio::runtime::task::harness::Harness<T,S>::poll::h2da67552fda7edb9
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  29:        0x154dcbfa0 - tokio::runtime::task::raw::poll::h2f6358036e21e2cf
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
  30:        0x154eb04b0 - tokio::runtime::task::raw::RawTask::poll::hafbff39549565f0b
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  31:        0x154eb944c - tokio::runtime::task::LocalNotified<S>::run::h246bca158db8637b
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
  32:        0x154eabfbc - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h6070f5802fb1ec23
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:576:13
  33:        0x154eabea0 - tokio::runtime::coop::with_budget::h7fc1cf45bd777a8a
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
  34:        0x154eabea0 - tokio::runtime::coop::budget::h2d07096d46097b77
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
  35:        0x154eabea0 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h28474f49af5b4a1a
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
  36:        0x154eab7ac - tokio::runtime::scheduler::multi_thread::worker::Context::run::h4b51da22208e01f3
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
  37:        0x154eab53c - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::hef4f7dbccbd300b4
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
  38:        0x154f16248 - tokio::runtime::context::scoped::Scoped<T>::set::h48967c3e1e3c9416
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
  39:        0x154ef7318 - tokio::runtime::context::set_scheduler::{{closure}}::ha85412cf3c5d248c
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26
  40:        0x154e989c8 - std::thread::local::LocalKey<T>::try_with::hf6966b26ac81f482
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/local.rs:270:16
  41:        0x154e95144 - std::thread::local::LocalKey<T>::with::hcf8f7976ac6e8c37
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/local.rs:246:9
  42:        0x154ef725c - tokio::runtime::context::set_scheduler::h4e98e3ae7b2ac870
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:9
  43:        0x154eab460 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h9ef275b241e76932
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
  44:        0x154f1aa3c - tokio::runtime::context::runtime::enter_runtime::h2dad1aac06750521
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
  45:        0x154eab2a0 - tokio::runtime::scheduler::multi_thread::worker::run::h0785b03457e9312c
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
  46:        0x154eab14c - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::hfc6e621eac9a0a4b
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
  47:        0x154ee4eec - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h9de3dca6c6d00643
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
  48:        0x154e5d940 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hfec594de548332cb
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  49:        0x154e5b7ac - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hb93a0dbdba224e7a
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  50:        0x154e5b7ac - tokio::runtime::task::core::Core<T,S>::poll::hd25500e57906fe49
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
  51:        0x154e50a60 - tokio::runtime::task::harness::poll_future::{{closure}}::h96d98bbba712eade
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  52:        0x154edb944 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h531849d3adde8f5f
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
  53:        0x154e36c04 - std::panicking::try::do_call::h1ec80eed4e700941
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
  54:        0x154e3db94 - ___rust_try
  55:        0x154e3102c - std::panicking::try::h08e79c44f7072e87
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  56:        0x154efff28 - std::panic::catch_unwind::h35fa19fb2c869dfb
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  57:        0x154e4d884 - tokio::runtime::task::harness::poll_future::h739e178f11ff8fd0
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  58:        0x154e417a8 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h3721f6c38aeae65d
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  59:        0x154e409c4 - tokio::runtime::task::harness::Harness<T,S>::poll::hb8fa6cc544bf5888
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  60:        0x154eb0b00 - tokio::runtime::task::raw::poll::h82f7e00145ca6939
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
  61:        0x154eb04b0 - tokio::runtime::task::raw::RawTask::poll::hafbff39549565f0b
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  62:        0x154eb9548 - tokio::runtime::task::UnownedTask<S>::run::h325b97c52029e9a1
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
  63:        0x154e8c928 - tokio::runtime::blocking::pool::Task::run::haf89b44256125981
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
  64:        0x154e94000 - tokio::runtime::blocking::pool::Inner::run::h861cf3bdaec96f81
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
  65:        0x154e93db4 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h4d4597093fed2b6e
                               at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13
  66:        0x154f25184 - std::sys_common::backtrace::__rust_begin_short_backtrace::h5e5ad456e93498fa
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:155:18
  67:        0x154edae78 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h023866687c51ede6
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:529:17
  68:        0x154edb814 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h46685ce984f28085
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
  69:        0x154e3959c - std::panicking::try::do_call::hf5618dba7a87df90
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
  70:        0x154e3db94 - ___rust_try
  71:        0x154e343a0 - std::panicking::try::h9a5be4bdbb5cc6aa
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  72:        0x154edacfc - std::panic::catch_unwind::hbd10baa2f7d47eec
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  73:        0x154edacfc - std::thread::Builder::spawn_unchecked_::{{closure}}::hf82a72b81ac2d572
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:528:30
  74:        0x154e6d0a8 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h30f9d5df81b8dbf0
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:250:5
  75:        0x155bf7d28 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h122931d29f919766
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  76:        0x155bf7d28 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h222f385c86e71c13
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  77:        0x155bf7d28 - std::sys::unix::thread::Thread::new::thread_start::h8dda2af345bab446
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  78:        0x181b8af94 - __pthread_joiner_wake
cjdsellers commented 1 month ago

TL;DR: I think we can fix these with a few changes to allow zero intervals and stop times in the past (which will just immediately produce a time event and expire the timer), making the live timer more robust and flexible.

For the first one, when setting an alert we still use a timer under the hood - and just set a stop time. For a very short duration it's possible for the stop time to become less than the time now. Consider this calculation where for the above scenario the interval_ns would become zero. (resulting in an immediate alert, which I think would be the expected result?)

alert_time_ns = std::cmp::max(alert_time_ns, ts_now);
let interval_ns = (alert_time_ns - ts_now).into();

Rather than this being an error condition which is very easy to run into, we can just relax the validation that the interval has to be positive (so its still possible to pass in ridiculous values such a 1, as before).

For the other error, its a similar scenario and would just result in an immediate timer expiry - which is fine. So I've also removed that condition check b7373ffea9b64f7bcaad7aeee148da7d54adf452.

One really bad error I can see by using an unsigned int for interval is if anyone passed in a negative interval_ns (which wouldn't happen with a time alert, but could happen with a timer), it's going to underflow back around to set some enormous interval the timer would never reach.

cjdsellers commented 1 month ago

Let me know how this goes on your sandbox run :pray:.

davidsblom commented 1 month ago

Awesome! Thanks for making the changes. Just started a new run. Will report later today how it went.

davidsblom commented 1 month ago

Running for 2 hours now without any warnings, errors or panics. Very nice thus far!

davidsblom commented 1 month ago

Still running fine. I think it is safe to assume that all issues are resolved with the sandbox.