Closed emakkus closed 1 year ago
is this happening only on llama2-based models? I have been using also contexts up to 8k just fine here with a GPU. Also - can you try master? I believe there were couple of fixes wrt to Llama2 that should probably matter here
@mudler Never mind... I think this is a problem on my end... I tried to run the same model on LLamaSharp and got the same segfault at around 500 Tokens. I'll try out other models but this pretty much proves that the problem is not on the LocalAI side. Thx anyway :)
LocalAI version:
1.23.2-cuda-12
Environment, CPU architecture, OS, and Version:
Linux lxdocker 6.2.16-4-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-5 (2023-07-14T17:53Z) x86_64 x86_64 x86_64 GNU/Linux Proxmox lxc with Docker running inside it.
Describe the bug Even though the model is configured to accept a context size of well beyond 512 Tokens, the inference still crashes if the prompt exceeds the limit of about 500 Tokens.
This is a problem for Chat conversations, since this limit is reached quite fast when older messages are added to the prompt for conversation context awareness of the AI.
To Reproduce
Simply try to call the chat/completion endpoint of a llama2 model with a prompt that has more than about 500 Tokens.
In my case, I can reproduce the Error with the following cURL:
Using this hacked together curl will result in the error being thrown. Removing just one word however will make this cURL work.
Expected behavior
The limit should be the one set in the model configuration. If I set the context size to 4096, I would expect to be able to use bigger prompts
Logs
Additional context
This is my docker-compose file I use to spin up localAi. I also overwrite the base.yaml to set additional configs:
This is the newhope config that's being created:
I absolutely love this project, and I hope there is a solution for this problem. If I'm missing a config option, please point it out to me. However the logs seem to indicate that the context size in the config is recognized.
Btw I'm using the following model from TheBloke: https://huggingface.co/TheBloke/NewHope-GGML
But I have tried other models, removed the rope config stuff but nothing seems to work. Once the 500 Token-Count is breached the Call crashes.