search

mudler / LocalAI

:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video, Images, Voice Cloning, Distributed inference
https://localai.io
MIT License
24.31k stars 1.86k forks source link

llamacpp chat/completions response unrelated to prompt on cpu local deploy #1918

Closed semsion closed 7 months ago

semsion commented 7 months ago

OS: Ubuntu 23.04. CPU: Intel i7-11370H 4.8Ghz (x8) RAM: 32GB

Via a local deployment, when calling the chat/completions endpoint via llamacpp, with the Docker AIO image, and a basic prompt, an unexpected response is being receiving. This has happened repeatedly over multiple tries.

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "bakllava.gguf", "messages": [{"role": "user", "content": "Hello", "temperature": 0.9}] }'
{"created":1711583083,"object":"chat.completion","id":"bdffe1ef-5cff-4846-86ea-dce4fffb64bc","model":"bakllava.gguf","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":" Theophilus and all the saints. In the Name of Our Lord and Saviour, Jesus Christ. Amen.\n Teil III.  5.  St. Theophilus of Salamis.  114-1155.  St. Theophilus.  Salamis.  Cyprus.  St. Irenaeus.  Lyons. 115.  St. Theophilus.  Salamis.  Cyprus.  St. Irenaeus.  Lyons. 115.  St. Theophilus.  Salamis.  Cyprus.  St. Irenaeus.  Lyons. 115.  St. Theophilus.  Salamis.  Cyprus.  St. Irenaeus.  Lyons. 115.  St. Theophilus.  Salamis.  Cyprus.  St. Irenaeus.  Lyons. 115.  St. Theophilus.  Salamis.  Cyprus.  St. Irenaeus.  Lyons. 115.  St. Theophilus.  Salamis.  Cyprus.  St. Irenaeus.  Lyons. 115.  St. Theophilus.  Salamis.  Cyprus.  St. Irenaeus.  Lyons. 115.  St. Theophilus.  Salamis.  Cyprus.  St. Irenaeus.  Lyons. 115.  St. Theophilus.  Salamis.  Cyprus.  St. Irenaeus.  Lyons. 115.  St. Theophilus.  Salamis.  Cyprus.  St. Irenaeus.  Lyons. 115.  St. Theophilus.  Salamis.  Cyprus.  St. Irenaeus.  Lyons. 115.  St. Theophilus.  Salamis.  Cyprus.  St. Irenaeus.  Lyons. 115."}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

Does anyone have any information by this could possibly be happening?!

mudler commented 7 months ago

Don't use the model file unless you want to handle the prompt template for yourself.

Just use the model names like you would do with OpenAI. For instance gpt-4-vision-preview, or gpt-4 are already present in the AIO images