Ollama Network Error

[sergiogcharles]

Hi, I configured Danswer to use Ollama, running ollama run llama2, and verified that the api server responds with curl. It works well in cli, but when I prompt the model in Danswer I see network error. I looked in the ngnix container logs, but there don't seem to be any errors. I read that this might be a problem with ngnix timeouts, so I tried to change app.conf.template.dev to this, but still no luck:

server {

    listen 80;

    server_name ${DOMAIN};

    client_max_body_size 5G;    # Maximum upload size 

    proxy_connect_timeout 600;

    proxy_send_timeout 600;

    proxy_read_timeout 600;

    send_timeout 600;

....
}

These are the api server's logs:

api_server-1     | 03/01/2024 09:49:06 PM      chat_backend.py 174 : Received new chat message: TEST

api_server-1     | INFO:     172.20.0.7:54974 - "POST /chat/send-message HTTP/1.1" 200 OK

api_server-1     | failed to send, dropping 1 traces to intake at http://localhost:8126/v0.5/traces after 3 retries

nginx-1          | 172.20.0.1 - - [01/Mar/2024:21:49:06 +0000] "GET /chat?chatId=25&_rsc=ts8bv HTTP/1.1 " 200 128 "http://localhost:3000/chat?chatId=43 " "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 " "- "

background-1     | [2024-03-01 21:49:08,112: INFO/MainProcess] Scheduler: Sending due task check-for-document-set-sync (check_for_document_sets_sync_task)

background-1     | [2024-03-01 21:49:08,467: INFO/MainProcess] Task check_for_document_sets_sync_task[73a953d9-1d5c-4f5d-a473-3b3aa82e8e98] received

background-1     | [2024-03-01 21:49:08,478: INFO/MainProcess] Task check_for_document_sets_sync_task[73a953d9-1d5c-4f5d-a473-3b3aa82e8e98] succeeded in 0.010725432002800517s: None

background-1     | [2024-03-01 21:49:13,112: INFO/MainProcess] Scheduler: Sending due task check-for-document-set-sync (check_for_document_sets_sync_task)

background-1     | 03/01/2024 09:49:13 PM            update.py 446 : Running update, current UTC time: 2024-03-01 21:49:13

background-1     | 03/01/2024 09:49:13 PM            update.py 315 : Found 0 new indexing tasks.

background-1     | [2024-03-01 21:49:13,523: INFO/MainProcess] Task check_for_document_sets_sync_task[d67ce699-1d33-48ed-9e20-d29a7c72c81a] received

background-1     | [2024-03-01 21:49:13,570: INFO/MainProcess] Task check_for_document_sets_sync_task[d67ce699-1d33-48ed-9e20-d29a7c72c81a] succeeded in 0.046063574991421774s: None

background-1     | [2024-03-01 21:49:18,112: INFO/MainProcess] Scheduler: Sending due task check-for-document-set-sync (check_for_document_sets_sync_task)

background-1     | [2024-03-01 21:49:18,562: INFO/MainProcess] Task check_for_document_sets_sync_task[ee146599-8459-4746-ab8a-d0f00de2be2f] received

background-1     | [2024-03-01 21:49:18,649: INFO/MainProcess] Task check_for_document_sets_sync_task[ee146599-8459-4746-ab8a-d0f00de2be2f] succeeded in 0.08630300099321175s: None

background-1     | [2024-03-01 21:49:23,112: INFO/MainProcess] Scheduler: Sending due task check-for-document-set-sync (check_for_document_sets_sync_task)

background-1     | 03/01/2024 09:49:23 PM            update.py 446 : Running update, current UTC time: 2024-03-01 21:49:23

background-1     | 03/01/2024 09:49:23 PM            update.py 315 : Found 0 new indexing tasks.

background-1     | [2024-03-01 21:49:23,617: INFO/MainProcess] Task check_for_document_sets_sync_task[383ca17e-e2e7-45dc-a045-55b7a632daa2] received

background-1     | [2024-03-01 21:49:23,653: INFO/MainProcess] Task check_for_document_sets_sync_task[383ca17e-e2e7-45dc-a045-55b7a632daa2] succeeded in 0.03628901000774931s: None

background-1     | [2024-03-01 21:49:28,112: INFO/MainProcess] Scheduler: Sending due task check-for-document-set-sync (check_for_document_sets_sync_task)

api_server-1     | 03/01/2024 09:49:29 PM            timing.py  71 : stream_chat_message took 23.469226837158203 seconds

nginx-1          | 172.20.0.1 - - [01/Mar/2024:21:49:29 +0000] "POST /api/chat/send-message HTTP/1.1 " 200 287 "http://localhost:3000/chat?chatId=43 " "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 " "- "

background-1     | [2024-03-01 21:49:29,663: INFO/MainProcess] Task check_for_document_sets_sync_task[335ab539-9ef9-431d-9697-79e7bf59f9da] received

background-1     | [2024-03-01 21:49:29,872: INFO/MainProcess] Task check_for_document_sets_sync_task[335ab539-9ef9-431d-9697-79e7bf59f9da] succeeded in 0.20813776900467928s: None

background-1     | [2024-03-01 21:49:33,112: INFO/MainProcess] Scheduler: Sending due task check-for-document-set-sync (check_for_document_sets_sync_task)

As you can see, I'm getting api_server-1 | failed to send, dropping 1 traces to intake at http://localhost:8126/v0.5/traces after 3 retries.

Thanks!

[useruid]

@sergiogcharles , hello!

On local machine I use this configuration in .env (using docker-compose-dev)

But when I run the same on VM into enterprise net I use like this: GEN_AI_API_ENDPOINT=YOUR_VM_IP:11434

Also i run ollama in docker with GPU boost https://ollama.com/blog/ollama-is-now-available-as-an-official-docker-image

Probably this can help you...

[sergiogcharles]

@sergiogcharles , hello!

On local machine I use this configuration in .env (using docker-compose-dev)

But when I run the same on VM into enterprise net I use like this: GEN_AI_API_ENDPOINT=YOUR_VM_IP:11434

Also i run ollama in docker with GPU boost https://ollama.com/blog/ollama-is-now-available-as-an-official-docker-image

Probably this can help you...

Hey, thanks for the response. Unfortunately, I am using this exact configuration, but I'm still seeing a network error.

[chemista]

@sergiogcharles I have the same issue with Ollama. Tag v0.3.36 is working fine, so reverting to that is a quickfix. Maybe bumping litellm in v.0.3.37 introduced a bug.

[sergiogcharles]

3.36

Hi @chemista, did you have to change anything besides creating the .env config? Also, are you running ollama from within a docker container? Thanks!

[chemista]

@sergiogcharles only .env config. I'm running Ollama via docker on the same machine. Just remember to delete docker volumes from the buggy version before you start the v0.3.36 with docker-compose.

[sergiogcharles]

@chemista Still no luck, unfortunately. I'll try some other tags and see if I can get it to work. Thanks for your help!

[sergiogcharles]

@sergiogcharles only .env config. I'm running Ollama via docker on the same machine. Just remember to delete docker volumes from the buggy version before you start the v0.3.36 with docker-compose.

@chemista I was curious if you're running Docker with the WSL 2 based engine?

[sergiogcharles]

@chemista I got it working! Note to future users: make sure you enable the 'Add the *.docker.internal names' option if you're using the WSL2 based engine. I am using tag 0.3.23.

[yuhongsun96]

Hey folks, I've found an issue that is likely causing this for you all. Some arguments which LiteLLM were suppose to drop when not applicable weren't being dropped correctly and was causing Ollama to have some very weird behavior and failing. Hopefully the latest tag works for you all now!

Guide also updated: https://docs.danswer.dev/gen_ai_configs/ollama

Will just mention though that the Ollama model options are still worse than the OpenAI options. If possible, we suggest using GPT4 or GPT4-Turbo. With the guide above, a lot of features are turned off impacting the quality of the experience.

[janantala]

Hello, I have same error failed to send, dropping 1 traces to intake at http://localhost:8126/v0.5/traces after 3 retries, 1 additional messages skipped with azure open ai (gpt4)

[al-lac]

Hey! With the latest version i also get the network error when using llama2 or llama3. If i just chat with the GPT directly it gives me a response. Just the danswer and paraphrase chats are having this issue.

OpenAI works just fine.