Krakonos
commented
2 months ago Ok, so I made a bit of a progress. I found out the open webui is not really openai compatible as I thought. Not sure where I got that impression. I can use the /ollama endpoint to authenticate using a token and pass down my query. Right now, it boils down to:
url = "http://192.168.0.61:8080/ollama", --doesn't work
url = "http://192.168.0.61:11434", -- worksBut hitting the API from a script seems to return exactly the same response:
#!/usr/bin/bash
curl 'http://192.168.0.61:8080/ollama/api/generate' \
-H 'accept: */*' \
-H 'authorization: Bearer sk-182affc34e0e43218580222cdd88ca06' \
-H 'content-type: application/json' \
--data-raw "{\"model\": \"codellama:7b-code\", \"options\": {\"temperature\":0.2, \"top_p\": 0.95}, \"prompt\":\"<PRE> #include <iostream>\\n#include <memory>\\n\\n\\n\\nclass A {\\n\\n};\\n\\nclass B {\\n\\n};\\n\\nint foo() {\\n\\n}\\n\\nint main() {\\n std::shared_ptr<A> a = std::make_shared<A>();\\n std::shared_ptr<B> b = std::make_shared<B>();\\n std::cout << \\\"Hello, World!\\\" << std::endl;\\n std::cout << a << std::endl;\\n std::cout << b << std::endl;\\n <SUF>\\n\\n\\n\\n\\n\\n\\n return 0;\\n\\n}\\n <MID>\", \"stream\": false}"
echo ""
echo "===="
echo ""
curl 'http://192.168.0.61:11434/api/generate' \
-H 'accept: */*' \
-H 'authorization: Bearer sk-182affc34e0e43218580222cdd88ca06' \
-H 'content-type: application/json' \
--data-raw "{\"model\": \"codellama:7b-code\", \"options\": {\"temperature\":0.2, \"top_p\": 0.95}, \"prompt\":\"<PRE> #include <iostream>\\n#include <memory>\\n\\n\\n\\nclass A {\\n\\n};\\n\\nclass B {\\n\\n};\\n\\nint foo() {\\n\\n}\\n\\nint main() {\\n std::shared_ptr<A> a = std::make_shared<A>();\\n std::shared_ptr<B> b = std::make_shared<B>();\\n std::cout << \\\"Hello, World!\\\" << std::endl;\\n std::cout << a << std::endl;\\n std::cout << b << std::endl;\\n <SUF>\\n\\n\\n\\n\\n\\n\\n return 0;\\n\\n}\\n <MID>\", \"stream\": false}"
#--data-raw '{"stream":false,"model":"codellama:7b-code","messages":[{"role":"user","content":"who are you?"}]}'(I copied the API endpoint URL and query body from llm-ls log). For some reason, when going through webui I get the unexpected error. Looking at the source, it appears that ollama backend does not pass down the api token, even if provided. This limits the use of open webui, which does a great job securing the ollama instance and allowing me to use my selfhosted llm server when traveling.
Hi!
I'm trying to set up llm-ls via llm.nvim plugin and I'm hitting weird serdes errors. I set the following config:
Weird thing is, completion seems to work just fine. But after each completion, I get a serdes error per following log. Any idea what am I doing wrong? I find it hard to find docs on using open webui as a backend, as well as the open webui docs for their API (I believe it is supposed to be openai compatible, hence I set it so).
Relevant logs (api key stripped):
... If you've got suggestions on a better alternative to open webui, I might consider it. But I want to open the server to a few friends and don't want the hassle of manually managing api keys for raw ollama instance (that works fine btw.).
Any help would be appreciated!