Ollama microservice failing KeyError: 'opea_service@llm_ollma'

[dhandhalyabhavik]

After building ollama image,

When tried running it, it failed with following error,

$ docker run --network host opea/llm-ollama:latest
/usr/local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:149: UserWarning: Field "model_name_or_path" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
[2024-06-21 02:55:49,512] [    INFO] - CORS is enabled.
[2024-06-21 02:55:49,513] [    INFO] - Setting up HTTP server
[2024-06-21 02:55:49,514] [    INFO] - Uvicorn server setup on port 9000
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:9000 (Press CTRL+C to quit)
[2024-06-21 02:55:49,528] [    INFO] - HTTP server setup successful
Traceback (most recent call last):
  File "/home/user/comps/llms/text-generation/ollama/llm.py", line 48, in <module>
    opea_microservices["opea_service@llm_ollma"].start()
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'opea_service@llm_ollma'

[feng-intel]

Runing on Xeon, I can reproduce the same issue. (And ollama service uses the same port 9000 with tgi service.)

[feng-intel]

Assign to Lvliang to help fix. Thanks.

[lvliang-intel]

already fixed by PR https://github.com/opea-project/GenAIComps/pull/248 @dhandhalyabhavik @feng-intel

[dhandhalyabhavik]

Hey @lvliang-intel, Thanks for fixing the issue. I am able to run llama-3.

but not able to run other models because llama-3 is hardcoded at line 24 here

Can you please enable it as REST argument similar to how ollama has?

like this

curl http://localhost:11434/api/generate -d '{
  "model": "llama3",
  "prompt":"Why is the sky blue?"
}'

[feng-intel]

HI @lvliang-intel Do we need to modify this hardcode here

@dhandhalyabhavik You can try to update "llama3" to other model.

[dhandhalyabhavik]

Then I will have to modify docker images every time I want to try new model, it should be just REST argument as provided above.

[lvliang-intel]

@dhandhalyabhavik， PR https://github.com/opea-project/GenAIComps/pull/280 fixed this issue.

[dhandhalyabhavik]

Thank you. Closing this now.