bug: thread 'notify-rs poll loop' panicked

[francis2tm]

Introduction

After a fresh install, sometimes edgen crashes before and after downloading default chat/completions model.

Using nix env.

Edit: It is not deterministic.

Log

thread 'notify-rs poll loop' panicked at crates/edgen_rt_llama_cpp/src/lib.rs:123:30:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
stack backtrace:
   0: rust_begin_unwind
             at /rustc/b66b7951b9b4258fc433f2919e72598fbcc1816e/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/b66b7951b9b4258fc433f2919e72598fbcc1816e/library/core/src/panicking.rs:72:14
   2: tokio::task::spawn::spawn_inner::panic_cold_display
   3: tokio::task::spawn::spawn
   4: <edgen_rt_llama_cpp::LlamaCppEndpoint as core::default::Default>::default
   5: once_cell::imp::OnceCell<T>::initialize::{{closure}}
   6: once_cell::imp::initialize_or_wait
   7: once_cell::imp::OnceCell<T>::initialize
   8: futures_executor::local_pool::block_on
   9: edgen_server::run_server::{{closure}}::{{closure}}
  10: <edgen_core::settings::UpdateHandler as notify::EventHandler>::handle_event
  11: notify::poll::data::WatchData::rescan
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
zephyr-7b-beta.Q4_K_M.gguf [00:03:39] [████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████] 4.07 GiB/4.07 GiB 18.97 MiB/s (0s)thread 'tokio-runtime-worker' panicked at ~/.cargo/registry/src/index.crates.io-6f17d22bba15001f/once_cell-1.19.0/src/lib.rs:1311:25:
Lazy instance has previously been poisoned
stack backtrace:
   0: rust_begin_unwind
             at /rustc/b66b7951b9b4258fc433f2919e72598fbcc1816e/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/b66b7951b9b4258fc433f2919e72598fbcc1816e/library/core/src/panicking.rs:72:14
   2: once_cell::imp::OnceCell<T>::initialize::{{closure}}
   3: once_cell::imp::initialize_or_wait
   4: once_cell::imp::OnceCell<T>::initialize
   5: <F as axum::handler::Handler<(M,T1),S>>::call::{{closure}}
   6: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
   7: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
   8: <tower::util::map_response::MapResponseFuture<F,N> as core::future::future::Future>::poll
   9: <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll
  10: <tower_http::cors::ResponseFuture<F> as core::future::future::Future>::poll
  11: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  12: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  13: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  14: <tower::util::map_response::MapResponseFuture<F,N> as core::future::future::Future>::poll
  15: <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll
  16: <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll
  17: hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T>::poll_catch
  18: <hyper::server::conn::http1::UpgradeableConnection<I,S> as core::future::future::Future>::poll
  19: <hyper_util::server::conn::auto::UpgradeableConnection<I,S,E> as core::future::future::Future>::poll
  20: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
  21: std::panicking::try
  22: tokio::runtime::task::harness::Harness<T,S>::poll
  23: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  24: tokio::runtime::context::scoped::Scoped<T>::set
  25: tokio::runtime::context::runtime::enter_runtime
  26: tokio::runtime::scheduler::multi_thread::worker::run
  27: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
  28: tokio::runtime::task::core::Core<T,S>::poll
  29: tokio::runtime::task::harness::Harness<T,S>::poll
  30: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'tokio-runtime-worker' panicked at ~/.cargo/registry/src/index.crates.io-6f17d22bba15001f/once_cell-1.19.0/src/lib.rs:1311:25:
Lazy instance has previously been poisoned
stack backtrace:
   0: rust_begin_unwind
             at /rustc/b66b7951b9b4258fc433f2919e72598fbcc1816e/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/b66b7951b9b4258fc433f2919e72598fbcc1816e/library/core/src/panicking.rs:72:14
   2: once_cell::imp::OnceCell<T>::initialize::{{closure}}
   3: once_cell::imp::initialize_or_wait
   4: once_cell::imp::OnceCell<T>::initialize
   5: <F as axum::handler::Handler<(M,T1),S>>::call::{{closure}}
   6: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
   7: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
   8: <tower::util::map_response::MapResponseFuture<F,N> as core::future::future::Future>::poll
   9: <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll
  10: <tower_http::cors::ResponseFuture<F> as core::future::future::Future>::poll
  11: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  12: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  13: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  14: <tower::util::map_response::MapResponseFuture<F,N> as core::future::future::Future>::poll
  15: <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll
  16: <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll
  17: hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T>::poll_catch
  18: <hyper::server::conn::http1::UpgradeableConnection<I,S> as core::future::future::Future>::poll
  19: <hyper_util::server::conn::auto::UpgradeableConnection<I,S,E> as core::future::future::Future>::poll
  20: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
  21: std::panicking::try
  22: tokio::runtime::task::harness::Harness<T,S>::poll
  23: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  24: tokio::runtime::context::scoped::Scoped<T>::set
  25: tokio::runtime::context::runtime::enter_runtime
  26: tokio::runtime::scheduler::multi_thread::worker::run
  27: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
  28: tokio::runtime::task::core::Core<T,S>::poll
  29: tokio::runtime::task::harness::Harness<T,S>::poll
  30: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Additionally, this bug also happens when resetting config through Tauri. Sometimes it works with no problem. Other times, it does not:

2024-01-30T23:24:29.127498Z  INFO edgen_core::settings: Creating new settings file: ~/.config/edgen/edgen.conf.yaml
2024-01-30T23:24:29.999864Z  INFO edgen_server: Thread has exited
2024-01-30T23:24:29.999875Z  INFO edgen_server: All threads have exited; exiting normally
thread 'notify-rs poll loop' panicked at crates/edgen_rt_llama_cpp/src/lib.rs:123:30:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
stack backtrace:
   0: rust_begin_unwind
             at /rustc/b66b7951b9b4258fc433f2919e72598fbcc1816e/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/b66b7951b9b4258fc433f2919e72598fbcc1816e/library/core/src/panicking.rs:72:14
   2: tokio::task::spawn::spawn_inner::panic_cold_display
   3: tokio::task::spawn::spawn
   4: <edgen_rt_llama_cpp::LlamaCppEndpoint as core::default::Default>::default
   5: once_cell::imp::OnceCell<T>::initialize::{{closure}}
   6: once_cell::imp::initialize_or_wait
   7: once_cell::imp::OnceCell<T>::initialize
   8: futures_executor::local_pool::block_on
   9: edgen_server::run_server::{{closure}}::{{closure}}
  10: <edgen_core::settings::UpdateHandler as notify::EventHandler>::handle_event
  11: notify::poll::data::WatchData::rescan
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2024-01-30T23:24:30.036796Z  INFO edgen_server: Settings have been updated, resetting environment
2024-01-30T23:24:30.036845Z  INFO edgen_server: Using default URI
2024-01-30T23:24:30.036982Z  INFO edgen_server: Listening in on: http://127.0.0.1:33322

[toschoo]

How often does it happen? 1 out of 10?

[toschoo]

What do you mean by "fresh install"? cargo build?

[toschoo]

update:

• First Bug: I was not able to reproduce it after more than 20 downloads
• Second Bug:
  • In my environment the bug always manifests on reset from tauri but also when saving the config file
  • it happens in and outside nix

[toschoo]

The second bug always occurs when one updates (or resets) the config file without having used the the lama model before. In that case the LlamaCppEndpoint is creates outside of a tokio runtime. I guess that happens because of a lazy initialisation, but I would like to understand better what happens.

Anyway, I have a very simple solution: when there is no runtime, create one and enter. That seems to work.

[toschoo]

For the second bug it's pretty clear, what happens:

• The LlamaCppEndpoint is initialised lazily to its default settings
• On a reset or update the llama environment is reset. Note that this code is executed in a synchronous closure where no runtime exists.
• The reset calls reset on the LlamaCppEndpoint which triggers the lazy initialisation and here, indeed, no runtime is available.