.Net: Test local modal with Semantic Kernel (i.e., Llama via Ollama)

[matthewbolanos]

Deploy a IChatCompletionService Llama model locally with Ollama and validate that it works with Semantic Kernel and the existing IChatCompletionService interface.

[matthewbolanos]

Doesn't need to test function calling.

[alliscode]

I tested our IChatCompletionService with Ollama using mistral. My conclusion is that our abstractions are good enough to allow this to work, but it does not currently work due to some implementation details in the Azure OpenAI SDK:

• Ollama chat API uses the same interface as OpenAI however the Ollama response is streaming by default where the OpenAI response is not-streaming by default. This is a problem because the only way to disable streaming on Ollama is to set streaming=false in the request body, which is never done in the Azure OpenAI SDK, it's either true or null (missing).
• Ollama streaming uses Server Sent Events just like OpenAI but Ollama uses named events, also known as multi-line events, see this and the Azure OpenAI SDK does not support this, it throws an exception.

Once these issues are fixed in the Azure OpenAI SDK, our IChatCompletionService will support Ollama.

[stephentoub]

Once these issues are fixed in the Azure OpenAI SDK, our IChatCompletionService will support Ollama.

Are there issues open on that for Azure.AI.OpenAI? Is anyone working on it? Timeframe?

[clement128]

hello any update for this?