make it more model-agnostic - through the kernel

[joslat]

The kernel is supporting more models, but we keep creating and doing everything by hand on the AgentBuilder (line 117 on AgentBuilder.cs for the PR https://github.com/microsoft/semantic-kernel/pull/3912/files) WithAzureOpenAIChatCompletion() for example. Similar to how the current version of SK is doing it: https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/Experimental/Agents/AgentBuilder.cs line 73, WithOpenAIChatCompletion.

But... the kernel supports already multi model so, let's use it shall we?

Suggestion: Add only a "WithKernel(Kernel kernel, string serviceId)" we could either copy the configuration of the kernel or clone the kernel (even more simple). I understand we want to decouple the kernel, but maybe we can reuse it if it allows parallel calls... If we look at the example AIServiceSelector, https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/KernelSyntaxExamples/Example62_CustomAIServiceSelector.cs

        var builder = Kernel.CreateBuilder()
            .AddAzureOpenAIChatCompletion(
                deploymentName: TestConfiguration.AzureOpenAI.ChatDeploymentName,
                endpoint: TestConfiguration.AzureOpenAI.Endpoint,
                apiKey: TestConfiguration.AzureOpenAI.ApiKey,
                serviceId: "AzureOpenAIChat",
                modelId: TestConfiguration.AzureOpenAI.ChatModelId)
            .AddOpenAIChatCompletion(
                modelId: TestConfiguration.OpenAI.ChatModelId,
                apiKey: TestConfiguration.OpenAI.ApiKey,
                serviceId: "OpenAIChat");

We could add a third or fourth local models, Ollama, LM Studio... through a service id. And then reuse the existing kernel for this (or create a new one if needed, unsure on the internals and if SK kernel allows calls in parallel)

[kbeaugrand]

@joslat I just released this week-end the support for Ollama... I'm taking a look to LL Studio

[joslat]

haha, I know, but on your private repo - I tried it this weekend but did not work (connect the ollama to the endpoint of LM Studio) ;)

[joslat]

Also, if it was not clear... what can I do to help? :)

[kbeaugrand]

I guess I could give en endpoint to configure the Kernel through Action<Kernel> ... What do you think about that?

[kbeaugrand]

haha, I know, but on your private repo - I tried it this weekend but did not work (connect the ollama to the endpoint of LM Studio) ;)

Yes it is on my private repos since I don't think MS team members will take my PR directly so, it won't be relevant to update the PR to match it until they ask me to do...

[joslat]

Do it! I know, but things are different if this is model-agnostic, leverages the "power of the central kernel" and matches the "Autogen flavor" - not? ;)

[kbeaugrand]

@joslat, I'm preparing an update in that direction. This would make easier to configure the kernel and its plugins by agents. It's also more convenient to add other connectors for the kernel, since they are not wrapped by the assistant builder.

Please see an example of how to configure the assistant with the new APIs:

string azureOpenAIEndpoint = configuration["AzureOpenAIEndpoint"]!;
string azureOpenAIDeploymentName = configuration["AzureOpenAIGPT35Endpoint"]!;
string azureOpenAIKey = configuration["AzureOpenAIAPIKey"]!;
string ollamaEndpoint = configuration["OllamaEndpoint"]!;

var financialKernel = Kernel.CreateBuilder()
                .AddAzureOpenAIChatCompletion(azureOpenAIDeploymentName, azureOpenAIEndpoint, azureOpenAIKey)
                .Build();

financialKernel.CreatePluginFromObject(new FinancialPlugin(), "financial");

var butlerKernel = Kernel.CreateBuilder()
                .AddOllamaChatCompletion("phi:latest", ollamaEndpoint)
                .Build();

var financialCalculator = AssistantBuilder.FromTemplate("./Assistants/FinancialCalculator.yaml")
    .WithKernel(financialKernel)
                .Build();

assistant = AssistantBuilder.FromTemplate("./Assistants/Butler.yaml",
       assistants: new IAssistant[]
       {
            financialCalculator
       })
    .WithKernel(butlerKernel)
    .Build();

[joslat]

@joslat, I'm preparing an update in that direction. This would make easier to configure the kernel and its plugins by agents. It's also more convenient to add other connectors for the kernel, since they are not wrapped by the assistant builder.

Please see an example of how to configure the assistant with the new APIs:

string azureOpenAIEndpoint = configuration["AzureOpenAIEndpoint"]!;
string azureOpenAIDeploymentName = configuration["AzureOpenAIGPT35Endpoint"]!;
string azureOpenAIKey = configuration["AzureOpenAIAPIKey"]!;
string ollamaEndpoint = configuration["OllamaEndpoint"]!;

var financialKernel = Kernel.CreateBuilder()
                .AddAzureOpenAIChatCompletion(azureOpenAIDeploymentName, azureOpenAIEndpoint, azureOpenAIKey)
                .Build();

financialKernel.CreatePluginFromObject(new FinancialPlugin(), "financial");

var butlerKernel = Kernel.CreateBuilder()
                .AddOllamaChatCompletion("phi:latest", ollamaEndpoint)
                .Build();

var financialCalculator = AssistantBuilder.FromTemplate("./Assistants/FinancialCalculator.yaml")
    .WithKernel(financialKernel)
                .Build();

assistant = AssistantBuilder.FromTemplate("./Assistants/Butler.yaml",
       assistants: new IAssistant[]
       {
            financialCalculator
       })
    .WithKernel(butlerKernel)
    .Build();

Why not create a single kernel with all the models... (less code) And just pass the "core kernel" with the service id. Like here: https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/KernelSyntaxExamples/Example61_MultipleLLMs.cs

For example, I kind of remade your code following the idea I have in mind (suggested by that example and the sk team I believe).

        Kernel kernel = Kernel.CreateBuilder()
            .AddAzureOpenAIChatCompletion(
                deploymentName: TestConfiguration.AzureOpenAI.ChatDeploymentName,
                endpoint: TestConfiguration.AzureOpenAI.Endpoint,
                apiKey: TestConfiguration.AzureOpenAI.ApiKey,
                serviceId: "AzureOpenAIChat",
                modelId: TestConfiguration.AzureOpenAI.ChatModelId)
            .AddOpenAIChatCompletion(
                modelId: TestConfiguration.OpenAI.ChatModelId,
                apiKey: TestConfiguration.OpenAI.ApiKey,
                serviceId: "OpenAIChat")
            .AddOllamaChatCompletion("phi:latest", ollamaEndpoint, "OllamaServiceId")
            .Build();

Agent1 = AgentBuilder.FromTemplate("./Assistants/Butler.yaml")
    .WithKernel(kernel, "AzureOpenAIChat")
    .Build();            

Agent2 = AgentBuilder.FromTemplate("./Assistants/Butler.yaml")
    .WithKernel(kernel, "OpenAIChat")
    .Build();     

Agent3 = AgentBuilder.FromTemplate("./Assistants/Butler.yaml")
    .WithKernel(kernel, "OllamaServiceId")
    .Build();     

And, inside the builder, it can request a clone of the kernel so each agent has its own kernel clone and can run in parallel.

How does this look to you? :)

Also, I'd rename AssistantBuilder to AgentBuilder, as Assistant is a term coming from the OpenAI Assistant API.