search

sobelio / llm-chain

`llm-chain` is a powerful rust crate for building chains in large language models allowing you to summarise text and complete complex tasks
https://llm-chain.xyz
MIT License
1.3k stars 128 forks source link

stop llama executor blocking #206

Closed andychenbruce closed 10 months ago

andychenbruce commented 1 year ago

If I understand correctly:

Before, the spawn_blocking was actually returning another function that was async which would then get awaited which undoes the whole purpose of the spawn_blocking.

The async doesn't work that well since it doesn't have any awaits other than the one locking the mutex, and I couldn't figure out how to get that to work in spawn_blocking so now it's changed to blocking_lock.

Also now join handle doesn't get awaited so if the thread panics it doesn't really handle it (I think the mutex lock can also get messed up with a panicked thread?), I don't know the proper way do deal with that.

Juzov commented 1 year ago

The changes makes sense to me. I think we should send_blocking in this case as well. Regaring the joinhandle yeah we're supposed to await it. This has? to occur at the end of the output stream somehow. Could the Output possibly hold the handle?

Juzov commented 10 months ago

On second thought LGTM