search

Mozilla-Ocho / llamafile

Distribute and run LLMs with a single file.
https://llamafile.ai
Other
20.53k stars 1.03k forks source link

Bug: llamafile completions API hangs on win10 #542

Closed Roeya closed 2 months ago

Roeya commented 3 months ago

Contact Details

roeyalmog@gmail.com

What happened?

I run llamafile load a model with or without --nobrowser flag (no meter which model I have tried) and start calling the API (none streamed). If for any reason I get a timeout many times the API stop responding, however the /health works just fine. Another problem that when this problem accrue I cannot use Ctrl+C to stop the process I must terminate it using the taskmanager.

Version

llamafile 0.8.2

What operating system are you seeing the problem on?

No response

Relevant log output

No response

jart commented 3 months ago

You're using a pretty old release. Could you try the latest llamafile 0.8.13?

jart commented 2 months ago

Closing due to no response. New release should fix it.

Roeya commented 1 month ago

I re tested it on llamafile 0.8.13 it still hangs - what is odd that the health API call works while the chat/completions api does nothing . After many experiments I am not sure if it relates to timeout