Open adut opened 6 months ago
Hello, looks like some things changed since the last time I tested with Ollama, brought things up to date and it should be working now. Guide also updated: https://docs.danswer.dev/gen_ai_configs/ollama
Will just mention though that the Ollama model options are still worse than the OpenAI options. If possible, we suggest using GPT4 or GPT4-Turbo. With the guide above, a lot of features are turned off impacting the quality of the experience.
Hi @yuhongsun96,
Could you explain why many features need to be disabled when using Ollama? If I understood correctly, based on the .env file mentioned in the documentation for Ollama, prompts are not properly optimized for models other than GPT-4, right? Would it be challenging to adjust these prompts for use with a self-hosted model? It would be great to be able to use Mixtral 8x7B, for instance, whose performance is comparable to GPT-3.5. Implementing these improvements would enable users to utilize Danswer entirely in an on-premises environment.
Thanks for your work!
I'm experience the same Ollama problem running the latest versions of Ollama and the latest danswer code. It's frustrating because the information is clearly right there in the search results below the text box. Periodically, I also seem to get three 500 errors in a row near the initial set of calls to Ollama:
[GIN] 2024/05/07 - 18:02:12 | 500 | 49.822208ms | 127.0.0.1 | POST "/api/generate" [GIN] 2024/05/07 - 18:02:12 | 500 | 68.797667ms | 127.0.0.1 | POST "/api/generate" [GIN] 2024/05/07 - 18:02:12 | 500 | 38.366917ms | 127.0.0.1 | POST "/api/generate"
I have have created multiple connectors, Web, Jira, Confluence, Local Files, Github and all were working with embedding and generating answers when using search, e.g "summarize X" or "explain Y" in v0.3.36.
Here is an example:
It would display the search results and an AI answer using embedding.
As of 0.3.37 it still shows the search results, but in AI answer it only shows "Information Not Found"
I am running the docker-compose deploy on my local machine.
Following is the output in my log file for the query, from the api-server, when it fails, not sure if that's helpful. Let me know if there are any other logs I can pull for you or information I can provide that will be helpfull: