⚡ Edgen: Local, private GenAI server alternative to OpenAI. No GPU required. Run AI models locally: LLMs (Llama2, Mistral, Mixtral...), Speech-to-text (whisper) and many others.
I guess that the problem is the lazy implementation of UnloadingModel but I couldn't prove it yet. The bug is simply too rare. If this is the problem, however, retry should solve the issue.
Remark
The code in Whisper is similar and, whatever solution is found for LLM, it should also be applied there.
Description
This line, very sporadically, causes a panic.
Solution
I guess that the problem is the lazy implementation of
UnloadingModel
but I couldn't prove it yet. The bug is simply too rare. If this is the problem, however, retry should solve the issue.Remark
The code in Whisper is similar and, whatever solution is found for LLM, it should also be applied there.