ollama AssertionError for cli/python

[Mikkicon]

I followed a zero_to_hero_guide and am facing this issue for

llama_models==0.0.54
llama_stack==0.0.54
llama_stack_client==0.0.54
MODEL_NAME = "meta-llama/Llama-3.2-3B-Instruct"
OLLAMA_INFERENCE_MODEL = "llama3.2:3b-instruct-fp16"

llama-stack-client

$ llama-stack-client --endpoint http://localhost:5001 inference chat-completion --message "hello, what model are you?"

Failed to inference chat-completion                                                          
Error Type: InternalServerError                                                              
Details: Error code: 500 - {'detail': 'Internal server error: An unexpected error occurred.'}

ollama docker

  File "/usr/local/lib/python3.10/site-packages/llama_stack/providers/remote/inference/ollama/ollama.py", line 257, in _nonstream_chat_completion
    assert isinstance(r, dict)
AssertionError

[ashwinb]

cc @init27 @heyjustinai just fyi if you see this more in the wild, please report. this seems odd and seems like we had tested this path a bunch!

[Mikkicon]

@ashwinb FYI it works for this endpoint, so it is possibly an Ollama issue

LLAMA_STACK_API_TOGETHER_URL="https://llama-stack.together.ai"

[ashwinb]

@Mikkicon can you tell me

• what machine you are running on? Is it a server? a laptop? Mac? Showing output of uname -a would be useful
• what is your exact command for running the llama stack ollama docker?