search

rustformers / llm

[Unmaintained, see README] An ecosystem of Rust libraries for working with large language models
https://docs.rs/llm/latest/llm/
Apache License 2.0
6.06k stars 350 forks source link

When using tokio and HuggingFaceRemote it breaks dropping the runtime #449

Open gadLinux opened 4 months ago

gadLinux commented 4 months ago

It seems that it's trying to connect to the outside but fails to do it so and it breaks. 

[2024-02-08T17:39:28Z INFO  main] "Starting assistant"
thread 'main' panicked at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/blocking/shutdown.rs:51:21:
Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.
stack backtrace:
   0: rust_begin_unwind
             at /rustc/7ffc697ce10f19447c0ce338428ae4b9bc0c041c/library/std/src/panicking.rs:647:5
   1: core::panicking::panic_fmt
             at /rustc/7ffc697ce10f19447c0ce338428ae4b9bc0c041c/library/core/src/panicking.rs:72:14
   2: tokio::runtime::blocking::shutdown::Receiver::wait
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/blocking/shutdown.rs:51:21
   3: tokio::runtime::blocking::pool::BlockingPool::shutdown
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/blocking/pool.rs:261:12
   4: <tokio::runtime::blocking::pool::BlockingPool as core::ops::drop::Drop>::drop
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/blocking/pool.rs:278:9
   5: core::ptr::drop_in_place<tokio::runtime::blocking::pool::BlockingPool>
             at /rustc/7ffc697ce10f19447c0ce338428ae4b9bc0c041c/library/core/src/ptr/mod.rs:507:1
   6: core::ptr::drop_in_place<tokio::runtime::runtime::Runtime>
             at /rustc/7ffc697ce10f19447c0ce338428ae4b9bc0c041c/library/core/src/ptr/mod.rs:507:1
   7: reqwest::blocking::wait::enter
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.23/src/blocking/wait.rs:76:21
   8: reqwest::blocking::wait::timeout
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.23/src/blocking/wait.rs:13:5
   9: reqwest::blocking::client::ClientHandle::new
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.23/src/blocking/client.rs:1075:15
  10: reqwest::blocking::client::ClientBuilder::build
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.23/src/blocking/client.rs:103:9
  11: cached_path::cache::CacheBuilder::build
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cached-path-0.6.1/src/cache.rs:130:27
  12: tokenizers::utils::from_pretrained::from_pretrained
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.13.4/src/utils/from_pretrained.rs:151:17
  13: tokenizers::tokenizer::Tokenizer::from_pretrained
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.13.4/src/tokenizer/mod.rs:445:30
  14: llm_base::tokenizer::TokenizerSource::retrieve
             at /home/gaguilar/.cargo/git/checkouts/llm-679a994ae16a26de/00aaf4a/crates/llm-base/src/tokenizer/mod.rs:116:17
  15: llm_base::loader::load
             at /home/gaguilar/.cargo/git/checkouts/llm-679a994ae16a26de/00aaf4a/crates/llm-base/src/loader.rs:382:21
  16: llm::loader::load
             at /home/gaguilar/.cargo/git/checkouts/llm-679a994ae16a26de/00aaf4a/crates/llm/src/loader.rs:26:5
  17: domain::services::build_model
             at ./domain/src/services/mod.rs:48:17
  18: <domain::services::AIModelProvider as core::default::Default>::default
             at ./domain/src/services/mod.rs:39:20
  19: infra::rest::router::build_router::{{closure}}
             at ./infra/src/rest/router.rs:50:25
  20: main::main::{{closure}}
             at ./api/src/bin/main.rs:31:51
  21: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/park.rs:281:63
  22: tokio::runtime::coop::with_budget
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/coop.rs:107:5
  23: tokio::runtime::coop::budget
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/coop.rs:73:5
  24: tokio::runtime::park::CachedParkThread::block_on
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/park.rs:281:31
  25: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/context/blocking.rs:66:9
  26: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
  27: tokio::runtime::context::runtime::enter_runtime
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/context/runtime.rs:65:16
  28: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
  29: tokio::runtime::runtime::Runtime::block_on
             at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/runtime.rs:350:45
  30: main::main
             at ./api/src/bin/main.rs:39:5
  31: core::ops::function::FnOnce::call_once
             at /rustc/7ffc697ce10f19447c0ce338428ae4b9bc0c041c/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
philpax commented 4 months ago

Hmm, that's quite odd. I don't think this is an issue with llm in particular; this looks to be more a strange interaction with tokenizers and your environment. (We don't explicitly do anything with async - it looks like tokenizers spins up a Tokio runtime to fetch the file and doesn't clean it up properly).

If this is an issue for your application, I'd suggest fetching the tokenizer or storing it yourself. The remote functionality is provided as a convenience, but it comes with a fair few caveats; this appears to be one of them 😦