Closed BongoUnicorns closed 3 weeks ago
I think it's caused by this bug in llama-cpp-python. Fix for that is already merged, so using version 0.2.86 or newer might fix it.
@kunesj is correct, it is related to the bug in llama-cpp-python. Upgrading the version to 0.2.88 in the conda prefix fixed it for me on v1.13; this version will need to be updated in the project
As this was resolved with 1.14, I'm closing this.
Describe the bug
When using llama.cpp loader and the latest oobabooga, any gguf model will generate text normally unless any grammar string is supplied, in which case the webui will crash entirely.
Logs attached have additional logs added into the code.
Is there an existing issue for this?
Reproduction
Clone latest webui and run setup. Load any gguf model using llama.cpp. Attempt generation normally. Add grammar string, webui will crash.
Screenshot
No response
Logs
System Info