Ollama

[morandalex]

How to config with ollama?

[hasparus]

Didn't get it to play nicely just yet, but maybe this can help the next person.

I'm at the following ~/.rawdog/config.yml contents.

llm_api_key: no need
llm_base_url: http://localhost:11434
llm_model: ollama/mistral

Error from rawdog:

Error:
 {'model': 'ollama/mistral', 'prompt': 'PROMPT: Hi', 'response': None, 'cost': None, 'error': 'Extra data: line 2 column 1 (char 92)'}
Error: Execution error: Extra data: line 2 column 1 (char 92)

Logs from ollama serve:

llama_new_context_with_model:      Metal compute buffer size =   171.60 MiB
llama_new_context_with_model:        CPU compute buffer size =     8.80 MiB
llama_new_context_with_model: graph splits (measure): 3
time=2024-02-03T08:45:44.901Z level=INFO source=dyn_ext_server.go:156 msg="Starting llama main loop"
[GIN] 2024/02/03 - 08:45:50 | 200 | 11.565958834s |       127.0.0.1 | POST     "/api/generate"
ggml_metal_free: deallocating

[kvaky]

This is caused by incompatibility between litelm and ollama, see recently added fix in litelm. In order to fix rawdog, just bump the litelm version in requirements to 1.22.3.

[tikendraw]

Config using litellm

1. Run litellm local server

   litellm --model ollama/phi --port 8080 --debug

2. Edit the .rawdog/config.yaml accordingly

   llm_api_key: AIxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxx
   llm_base_url: http://0.0.0.0:8080
   llm_custom_provider: openai
   llm_model: ollama/phi

[jakethekoenig]

I just edited the readme to me more descriptive. A simpler solution is to make your config the following:

llm_custom_provider: ollama
llm_model: mixtral