[BUG] - Ollama OpenAI not working

[bricklayer-co]

Description

When using LLM openai interface directly, everything else is available except for chat of GRAPHRAG Collection , which is not available. there is the error: /home/master/.local/lib/python3.10/site-packages/datashaper/engine/verbs/convert.py:72: UserWarning: Could not infer format, so each element will be parsed individually, falling back to dateutil. To ensure parsing is consisten File "/home/master/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run result = context.run(func, args) File "/home/master/.local/lib/python3.10/site-packages/gradio/utils.py", line 818, in wrapper response = f(args, **kwargs) File "/mnt/c/workplace/Pycharm-workplcae/kotaemon/libs/ktem/ktem/pages/chat/init.py", line 704, in message_selected return retrieval_history[index], plot_history[index] IndexError: list index out of range

But deploying local models qwen2 and nomic-embed-text using ollama,there is the error:

openai.APIConnectionError: Connection error.

Reproduction steps

1. The local model interface can be successfully called using OpenAI
2. The. env file has been set to ollama
'''
GRAPHRAG_API_KEY="ollama"
GRAPHRAG_API_BASE="http://localhost:11434/v1/"
GRAPHRAG_LLM_API_KEY="ollama"
GRAPHRAG_LLM_API_BASE="http://localhost:11434/v1/"
GRAPHRAG_LLM_MODEL="qwen2"
GRAPHRAG_EMBEDDING_API_KEY="ollama"
GRAPHRAG_EMBEDDING_API_BASE="http://localhost:11434/v1/"
GRAPHRAG_EMBEDDING_MODEL="nomic-embed-text"
OPENAI_API_BASE="http://localhost:11434/v1/"
OPENAI_API_KEY="ollama"
OPENAI_CHAT_MODEL="qwen2"
OPENAI_EMBEDDINGS_MODEL="nomic-embed-text"
'''
3. ./ktem_app_data/user_data/files/graphrag/xxxx/settings.yaml has been fixed using GRAPHRAG_LLM_MODEL, GRAPHRAG_EMBEDDING_MODEL by modifying the code
'''
llm:
  api_key: ${GRAPHRAG_API_KEY}
  type: openai_chat
  model: qwen2
  model_supports_json: true
  api_base: http://localhost:11434/v1/
parallelization:
  stagger: 0.3
async_mode: threaded
embeddings:
  async_mode: threaded
  llm:
    api_key: ${GRAPHRAG_API_KEY}
    type: openai_embedding
    model: nomic-embed-text
    api_base: http://localhost:11434/v1/
'''
4. According to local_madel.md, the configuration on the page has also been changed to ollama

Screenshots

No response

Logs

14:56:54,552 root ERROR error extracting graph
Traceback (most recent call last):
  File "/home/master/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
    yield
  File "/home/master/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 377, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/master/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
    raise exc from None
  File "/home/master/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
    response = await connection.handle_async_request(
  File "/home/master/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
    raise exc
  File "/home/master/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
    stream = await self._connect(request)
  File "/home/master/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 122, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
  File "/home/master/.local/lib/python3.10/site-packages/httpcore/_backends/auto.py", line 30, in connect_tcp
    return await self._backend.connect_tcp(
  File "/home/master/.local/lib/python3.10/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp
    with map_exceptions(exc_map):
  File "/home/master/anaconda3/envs/kotae/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/master/.local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: All connection attempts failed

Browsers

Chrome

OS

Windows

Additional information

No response

[gogodechen1]

"Why am I using this configuration:

makefile Copy code api_key: ollama base_url: http://localhost:11434/v1/ model: llama3.1:8b Test as follows:

Testing model: ollama Sending a message: Hi Connection failed. Got error: Error code: 400 - {'error': {'message': 'unexpected EOF', 'type': 'invalid_request_error', 'param': None, 'code': None}}. When testing with http://localhost:11434/ in the browser, it shows 'Ollama is running', but I get an error when setting it up. Why is this happening?"**

[yaboyang]

"Why am I using this configuration:

makefile Copy code api_key: ollama base_url: http://localhost:11434/v1/ model: llama3.1:8b Test as follows:

Testing model: ollama Sending a message: Hi Connection failed. Got error: Error code: 400 - {'error': {'message': 'unexpected EOF', 'type': 'invalid_request_error', 'param': None, 'code': None}}. When testing with http://localhost:11434/ in the browser, it shows 'Ollama is running', but I get an error when setting it up. Why is this happening?"**

I use the same setup, but the error is different.

• Connection failed. Got error: Error code: 404 - {'error': {'message': 'model "llama3.1:8b" not found, try pulling it first', 'type': 'api_error', 'param': None, 'code': None}}

And the same error insists even I changed the parameters. It seems it just don't read settings here.

And at the same time, the settings on Embeddings Tab (also ollama) is work

[yaboyang]

"Why am I using this configuration: makefile Copy code api_key: ollama base_url: http://localhost:11434/v1/ model: llama3.1:8b Test as follows: Testing model: ollama Sending a message: Hi Connection failed. Got error: Error code: 400 - {'error': {'message': 'unexpected EOF', 'type': 'invalid_request_error', 'param': None, 'code': None}}. When testing with http://localhost:11434/ in the browser, it shows 'Ollama is running', but I get an error when setting it up. Why is this happening?"**

I use the same setup, but the error is different.

• Connection failed. Got error: Error code: 404 - {'error': {'message': 'model "llama3.1:8b" not found, try pulling it first', 'type': 'api_error', 'param': None, 'code': None}}

And the same error insists even I changed the parameters. It seems it just don't read settings here.

And at the same time, the settings on Embeddings Tab (also ollama) is work

Guys, maybe u never find the real reason. Firstly, the settings on UI is not work at all. Secondly, it reports not found as I pull without version thus it's with tag latest. So I pull with :8b then it's successed, though there nothing need to download.

[munkito]

change model: llama3.1:8b to model: llama3.1:latest

[WeipengMO]

In the case of using Docker image, please replace http://localhost with http://host.docker.internal to correctly communicate with service on the host machine.

[Niko-La]

In the case of using Docker image, please replace http://localhost with http://host.docker.internal to correctly communicate with service on the host machine.

@WeipengMO nice!!!

also on docker

api_key: ollama
base_url: http://host.docker.internal:11434/v1/
model: phi3.5:latest

Have Ollama run phi3:latest in terminal left top Have Docker running fine able to get into Kotaemon

Not sure :crying_cat_face:

[WeipengMO]

Hi @Niko-La , I am running Docker on WSL2, following the steps outlined in this link under the "Use Local Models for RAG" section.

I hope this information helps.

[Niko-La]

@WeipengMO im on ubuntu so had to use the host's actual IP address instead of host.docker.internal.

and

Modify Ollama Configuration Ollama is running on the host machine, ensure it's listening on all interfaces, not just localhost. Add this to your Ollama service file:

Environment="OLLAMA_HOST=0.0.0.0:11434"

thx

nikola@nikola:~/Downloads/kotaemon$ ollama list
NAME                    ID              SIZE    MODIFIED       
nomic-embed-text:latest 0a109f422b47    274 MB  2 minutes ago   
gemma2:2b               8ccf136fdd52    1.6 GB  38 minutes ago  
phi3.5:latest           3b387c8dd9b7    2.2 GB  2 weeks ago     
llama3:instruct         a6990ed6be41    4.7 GB  4 months ago  

Local LLm and Embeddings now working using this setup @WeipengMO thx for your support