host_ip causes an error in ChatQnA AIPC deployment with Ollama

[devpramod]

Priority

P2-High

OS type

Ubuntu

Hardware type

AI-PC

Installation method

• [ ] Pull docker images from hub.docker.com
• [ ] Build docker images from source

Deploy method

• [X] Docker compose
• [ ] Docker
• [ ] Kubernetes
• [ ] Helm

Running nodes

Single Node

What's the version?

25174c0

Description

In GenAIComps, ollama is tested with localhost: curl http://localhost:11434/api/generate -d '{ "model": "llama3", "prompt":"Why is the sky blue?" }' ` curl http://127.0.0.1:9000/v1/chat/completions -X POST -d '{"model": "llama3", "query":"What is Deep Learning?","max_new_tokens":32,"top_k":10,"top_p":0.95,"typical_p":0.95,"temperature":0.01,"repetition_penalty":1.03,"streaming":true}' -H 'Content-Type: application/json'

` which works fine. But in GenAIExamples - instead of localhost, host_ip is used which causes an error (Connection refused)

Reproduce steps

cd GenAIExamples/ChatQnA/docker/aipc

docker compose up -d

run ollama

curl http://${host_ip}:9000/v1/chat/completions\ -X POST \ -d '{"query":"What is Deep Learning?","max_new_tokens":17,"top_k":10,"top_p":0.95,"typical_p":0.95,"temperature":0.01,"repetition_penalty":1.03,"streaming":true}' \ -H 'Content-Type: application/json'

Raw log

[No response](curl: (7) Failed to connect to x.x.x.x port 11434 after 0 ms: Connection refused)

[dcmiddle]

@devpramod can you add a meaningful title to the issue? Do you have host_ip set?

[devpramod]

@dcmiddle Yes all other services work fine with host_ip For Ollama, both the backend LLM service (i.e. Ollama itself) and the LLM microservice don't work with host_ip, need to set to localhost