Open clearstorm-tech opened 21 hours ago
Wondering if you have the content of your llama-stack-test-docker-run.yaml
? We should make sure that host/port should be pointed to the ollama endpoint for the ollama provider.
Sure, here it is.
version: v1
built_at: '2024-10-01T07:04:33.745095'
image_name: stack-test-docker
docker_image: stack-test-docker
conda_env: null
apis_to_serve:
- models
- memory_banks
- agents
- memory
- shields
- safety
- inference
api_providers:
inference:
providers:
- remote::ollama
memory:
providers:
- meta-reference
safety:
providers:
- meta-reference
agents:
provider_id: meta-reference
config:
persistence_store:
namespace: llama-stack
type: redis
host: localhost
port: 6379
telemetry:
provider_id: meta-reference
config: {}
routing_table:
inference:
- provider_id: remote::ollama
config:
host: localhost
port: 4001
routing_key: Llama3.1-8B-Instruct
memory:
- provider_id: meta-reference
config: {}
routing_key: vector
safety:
- provider_id: meta-reference
config:
llama_guard_shield: null
prompt_guard_shield: null
routing_key:
- llama_guard
- code_scanner_guard
- injection_shield
- jailbreak_shield
I've tried a few different ways to get this running and I'm not sure what I'm missing or if this is not working.
Running
After configuring with port 4001
router_api Api.inference router_api Api.safety router_api Api.memory Resolved 8 providers in topological order Api.models: routing_table Api.inference: router Api.shields: routing_table Api.safety: router Api.memory_banks: routing_table Api.memory: router Api.agents: meta-reference Api.telemetry: meta-reference
.... httpcore.ConnectError: All connection attempts failed
... RuntimeError: Ollama Server is not running, start it using
ollama serve
in a separate terminalThe thing is it is running and I have the models loaded and running. But this is on my host.
I've tried to set the host on the docker file with --add-host=host.docker.internal:host-gateway
I've also tried the conda build and get the same Ollama Server is not running error.
Is this a bug? or any insights from anyone would be great,
...or is this not meant to use the host machine local Ollama?