How to stream the chat from the Chat class?

[Sharaf-Mansour]

I am building an agentic workflow with Blazor and things were fine using the Ollama.StreamCompletion(), Thing is. I want to give my agents some roles and responsibilities. The only way you allowed this is by using Ollama.Chat() When I tried to parse out the message object that is returned and write it out. Which took a lot of effort things were not as smooth as Ollama.StreamCompletion as it contained both my Question and the AI answer, and I could not get clear stream. I also cannot save the context offline on my PC. because before I used System.Text.Json to store an offline Json file of the context in order to save the chat history. so I do not boot up my agents and build them from scratch every time I start the system. So, for me there is one of 2 fixes that could work. 1- Access both the context and the StreamCompletion from Ollama.Chat() So I can build my agents and stream their responses. 2- Have a way to assign roles or provide the Message object to the Ollama.StreamCompletion() because it already streams responses and store the context.

After taking a look at your code. I am sure that the 2nd option is the easiest.
My code using Chat

Action<ChatResponseStream> streamer = async stream =>
{
    stream.Message = new()
    {
        Role = ChatRole.System,
        Content = JrPrompt
    };
};
var JuniorChat =   Ollama.Chat(streamer);

My code using Stream Completion

 JunoirContext = await Ollama.StreamCompletion(JrPrompt, JunoirContext, async stream => Console.Write(stream.Response));

[awaescher]

I think I did not fully understand the issue you are facing. However, you're comparing StreamCompletion() and Chat() and that "I allow" things with one but not for both of them. Among all the endpoints Ollama offers, there are two different for streaming a completion and chatting:

• /api/generate which is StreamCompletion()
• /api/chat which is Chat().

So these different endpoints are designed for different purposes.

[Sharaf-Mansour]

@awaescher issue with generate I can not send a message object. which includes roles and content for example role: system, user and assistant And for context. I am building an agentic workflow. (2 AI talking to each other to complete some tasks) I want each one to have a persona. and store that persona offline. StreamCompletion() allows me to store the persona via context. but I cannot give it a system message. but with Chat() I can give it persona. But how will I even get back the response. Or store the context?

As a work around I switched to use Microsoft.SemanticKernel But I like your lib more because I can not store the Context there, So I send the whole chat to your OLLAMA lib to store the context and use both as a workaround which is in called Over Engineering

So, Is there a way to get the AI response from the chat() ? And store the context?? If not, Can I assign roles / send a system message using StreamCompletion()

[Sharaf-Mansour]

@awaescher Also note that the docs you sent for both. includes exactly what I am asking for.

HINT:

system: system message to (overrides what is defined in the Modelfile) in /api/generate

response A stream of JSON objects is returned in /api/chat

So, I am just asking for its implementation.

[awaescher]

If I get it right, you want to use the properties system, template and maybe more when using StreamCompletion(), is that correct?

Well, there should be an overload which takes an instance of the GenerateCompletionRequest.

The overload you are using is just an extension method that does exactly this in a convenient way that should use the defaults for 99% of the use cases.

[Sharaf-Mansour]

I have been reading the docs and could not find it, will give it a try now that you have mentioned it, Thank you @awaescher maybe it would be better if it was documented as well in the read me file ❤️❤️

[awaescher]

That's why I made the method and the extension method the same name so that people could find it without having to check the docs. Here's what it looks like:

[Sharaf-Mansour]

@awaescher alright, mind if I ask what is the real purpose of using the Chat() if it doesn't return the response or the context ? Or does it return them in a way that was not cleared in the docs?

[awaescher]

Once again, I have no idea whats missing.

Chat.Send() returns a ChatResponseStream instance that holds quite a few properties, including a Message instance which contains the following properties: Role (System/Assistant/User), Content and Images.

The Chat instance itself collects all the messages in an ongoing chat and exposes them with its Messages property. These also offer the properties mentioned above.

[Sharaf-Mansour]

That's why I made the method and the extension method the same name so that people could find it without having to check the docs

Small Question with StreamCompletion(). why did you use IResponseStreamer instead of Action? this is no longer valid

SeniorContext = Context = await Ollama.StreamCompletion( 
   new ()
   {
   System = SrPrompt,
   Context = SeniorContext.Context,
   Stream = true
   }, streamer:  stream => Console.Write(stream.Response));

Inconsistency is confusing here.

[Sharaf-Mansour]

@awaescher again I am asking because I do not even how or the right way to use this overloaded version of the function. What exactly to send and return. are they all inside the context? it is just a long[] very confusing.

The behavior and return of both version StreamCompletion() is not the same at all The expected behavior is to return the AI response and stream it. For both overloads of the function

I checked the implementation Understood the task build it from scratch :D

     SeniorContext = Context = await Ollama.StreamCompletion(new()
     {
         System = SrPrompt,
         Context = SeniorContext.Context,
         Stream = true

     }, new ActionResponseStreamer<GenerateCompletionResponseStream?>(new Action<GenerateCompletionResponseStream?>(

        async stream =>
        {
            await InvokeAsync(async () =>
            {
                Response += stream.Response;
                await InvokeAsync(StateHasChanged);
            }
            );
        }
        )));

[awaescher]

Great to see you got it working