search

danny-avila / LibreChat

Enhanced ChatGPT Clone: Features OpenAI, Assistants API, Azure, Groq, GPT-4 Vision, Mistral, Bing, Anthropic, OpenRouter, Vertex AI, Gemini, AI model switching, message search, langchain, DALL-E-3, ChatGPT Plugins, OpenAI Functions, Secure Multi-User System, Presets, completely open-source for self-hosting. More features in development
https://librechat.ai/
MIT License
15.28k stars 2.54k forks source link

[Bug]: Running LibreChat against LiteLLM backed by Ollama #1270

Closed netnem closed 7 months ago

netnem commented 7 months ago

Contact Details

No response

What happened?

I have Ollama/openchat running behind the OpenAPI compatible frontend of LiteLLM.

The chat completion never "finishes" when the bot is responding, appears to be searching for a chatcompletion message

Steps to Reproduce

install librechat via docker, in .env file set OPENAI_REVERSE_PROXY=http://192.168.2.142:8000 (i installed litellm on the docker host) install ollama via install script: curl https://ollama.ai/install.sh | sh pip install litellm ollama pull openchat pip install async_generator litellm --model ollama/openchat --api_base http://localhost:11434 --drop_params <-- required for presence penalty not being supported in litellm

The setup works for calling a localLLM, but the cursor gets "stuck" and does not return to the user to add additional chats.

What browsers are you seeing the problem on?

Firefox, Microsoft Edge

Relevant log output

There was an uncaught error:
OpenAIError: stream ended without producing a ChatCompletionMessage with role=assistant
    at ChatCompletionStream._AbstractChatCompletionRunner_getFinalMessage (/app/node_modules/openai/lib/AbstractChatCompletionRunner.js:464:11)
    at ChatCompletionStream._AbstractChatCompletionRunner_getFinalContent (/app/node_modules/openai/lib/AbstractChatCompletionRunner.js:455:134)
    at ChatCompletionStream._emitFinal (/app/node_modules/openai/lib/AbstractChatCompletionRunner.js:282:152)
    at /app/node_modules/openai/lib/AbstractChatCompletionRunner.js:77:22
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Clearing sync timeouts before exiting...

Screenshots

image

Code of Conduct

danny-avila commented 7 months ago

Thanks for your thorough report, hope to address this soon as I've seen this error in another context, and I can reproduce it now.

danny-avila commented 7 months ago

I'm fixing this right now

danny-avila commented 7 months ago

FYI I seem to have issues with ollama independent of LibreChat when I don't include --drop_params, related issue: https://github.com/BerriAI/litellm/issues/992#issuecomment-1839932916

netnem commented 7 months ago

Confirmed the latest merge works great.

Thanks @danny-avila