Treating Azure OpenAI Deployments as resources

[aaronpowell]

With the AOAI integration we can add a service resource into the AppHost like so:

var openai = builder.AddAzureOpenAI("openai")
                    .AddDeployment(new("gpt-35-turbo", "gpt-35-turbo", "0613"));

builder.AddProject<Projects.OpenAIEndToEnd_WebStory>("webstory")
       .WithReference(openai);

But within our client project we're going to need to know the name of the deployment to invoke it in the SDK:

    private ChatCompletionsOptions chatCompletionOptions = new ChatCompletionsOptions()
    {
        DeploymentName = "gpt-35-turbo",
        Messages =
        {
            new ChatRequestSystemMessage("Pick a random topic and write a sentence of a fictional story about it.")
        }
    };

Since the name of the deployment is any arbitrary string there's the possibility of it being miskeyed in the client application. My workaround approach to this is to have code like so:

.AddDeployment(new(builder.Configuration["Azure:AI:ChatDeploymentName"] ?? "gpt-4", "gpt-4", "1106"))

Where I use the config or fallback to a known default, but I then need to ensure that the config value is passed to the client using WithEnvironment.

It'd be convenient if there was a way that the deployment names were flowed into the client application, either allowing them to be created as parameters with AddParameter (and then passing that into the AzureOpenAIDeployment constructor) or when theAzureOpenAIResource` is passed to the projects as a resource the deployment info is also passed (probably as a different environment variable) - although the discovery of this could be challenging and it'd fall down if you're using a connection string for local dev like this:

var ai = builder.ExecutionContext.IsPublishMode ?
    builder.AddAzureOpenAI(ServiceNames.AzureOpenAI)
        .AddDeployment(new(builder.Configuration["Azure:AI:ChatDeploymentName"] ?? "gpt-4", "gpt-4", "1106"))
        .AddDeployment(new(builder.Configuration["Azure:AI:EmbeddingDeploymentName"] ?? "text-embedding-ada-002", "text-embedding-ada-002", "2")) :
    builder.AddConnectionString(ServiceNames.AzureOpenAI);

Since in this scenario the deployment info isn't available until provisioning happens.

[mitchdenny]

Looping in @tg-msft.

This is one of the areas where having "deep linked connection strings" are really helpful. When you define a database resource via Aspire like the following:

var builder = DistributedApplication.CreateBuilder(args);
var sqldb = builder.AddSqlServer("sql").AddDatabase("sqldb");
builder.AddProject<Projects.MyApp>("myapp")
       .WithReference(sqldb);

The connection string ConnectionStrings__sqldb will end up being "deep" in that it connects directly to the database, and the SqlConnection we inject ends up being pre-wired to the database.

With Open AI this isn't the case. We get a connection to the cog services account, but the deployment name and all other connection information is something that the developer needs to plumb through themselves. Aaron illustrates how this creates extra work because he is trying to come up with a creative way of defining the deployment name once and plumbing it through.

I'm a big proponent of two things:

1. Providing a single string that can be used to connect to a resource, and in the case where you need a child resource to actually do anything useful, identifiers for that should be on the connection string.
2. When the connection string is consumed, a resource should be injected into DI that is pre configured to point to that deeply linked location.

In the case of Azure Open AI I would like to see a connection string format defined that:

1. Specifies the endpoint
2. Specifies the deployment name
3. Specifies the authentication method
4. (optional) key if managed identity isn't used.

Then I would like to see an OpenAIDeploymentClient which is preconfigured to talk to a particular deployment.

[mitchdenny]

Glad you raised this @aaronpowell ... I was writing up an example of how you could plumb through the deployment name and found a bug. Fixing that now and will share a snippet of code with you in a sec.

[mitchdenny]

OK here is a code sample that I put together as part of this bug fix: #3092

var openai = builder.AddAzureOpenAI("openai", (_, _, _, deployments) => {
    var deployment = deployments.Single();
    deployment.AddOutput("modelName", x => x.Name);
}).AddDeployment(new("gpt-35-turbo", "gpt-35-turbo", "0613"));

builder.AddProject<Projects.OpenAIEndToEnd_WebStory>("webstory")
       .WithReference(openai)
       .WithEnvironment("OpenAI__DeploymentName", openai.GetOutput("modelName"));

[aaronpowell]

That's an interesting idea, using the outputs in that manner. I wonder if there's a way that we could automatically wire up the outputs across the pipeline

[mitchdenny]

My idea was silly and unncessary, just do this:

var deploymentAndModelName = "gpt-35-turbo";
var openai = builder.AddAzureOpenAI("openai").AddDeployment(
    new(deploymentAndModelName, deploymentAndModelName, "0613")
    );

builder.AddProject<Projects.OpenAIEndToEnd_WebStory>("webstory")
       .WithReference(openai)
       .WithEnvironment("OpenAI__DeploymentName", deploymentAndModelName);

[mitchdenny]

One of the benefits of Aspire is you can use plain old C# :)

[mitchdenny]

Proposing we close this if you are happy @aaronpowell

[aaronpowell]

While yes, that approach solves the problem (it's essentially what I do), I wonder if it can't be done is a more automatic manner. After all, you know that the deployments are going to be needed, or is there a concern about exposing info that isn't needed (say providing a deployment that isn't used by a particular service)?

I'm trying to think of other resources where you might want to bring in some additional data in this manner but I am struggling a bit to think of any, maybe it's best to leave AOAI as an outlayer.

[timheuer]

@mitchdenny could you have a WithReference overload and add them both in there, so that the reference gets the deployment automatically without more code?

[mitchdenny]

I think that gets messy pretty quickly. What if you have two OpenAI accounts both with the same deployment name. The variable that you inject for the deployment name now needs to include the OpenAI account resource name to make sure there isn't a conflict.

Then you have to consider the consuming side, how do they know what the configuration name is?

Lets imagine that AddDeployment didn't just flow through AzureOpenAIResource builder, and instead returned a deployment builder:

var builder = DistributedApplication.CreateBuilder(args);
var aoai = builder.AddAzureOpenAI("openai").AddDeployment("foo", ...);
var app = builder.AddProject<Projects.MyApp>("app")
                 .WithReference(aoai);

... the problem you are going to have is on the consumption side. How does the developer know what environment variable to get the deployment name out of. The AzureOpenAIClient doesn't allow you to create a client based on the deployment name, that is something you pass into chat completion options.

We've moved from a problem of the developer having to do something themselves to a developer having to guess what environment variable we stashed this key piece of data in.

The way to tackle these kinds of issues is go upstream and talk to the AI team about providing a connection string format that includes the deployment name, and having some kind of client that can have a pre-filled deployment name.

I suspect that horse has bolted though.

[aaronpowell]

The way to tackle these kinds of issues is go upstream and talk to the AI team about providing a connection string format that includes the deployment name, and having some kind of client that can have a pre-filled deployment name.

Problem with that is that the client is agnostic of the model, model is specified per-request, since the client is really just a typed HttpClient.

[aaronpowell]

I thought I'd bring this back up as there's been a bunch of changes in the space since April when it was last discussed.

New developments

In the Ollama integration for the Community Toolkit we've gone with the approach that the model is a resource, meaning you can provide the model resource rather than the Ollama resource as the reference:

var ollama = builder.AddOllama("ollama");

var phi35 = ollama.AddModel("phi3.5");

var exampleProject = builder.AddProject<Projects.ExampleProject>()
                            .WithReference(phi35);

This means that the client can get the right model without having to provide the connection name, ollama, and also the model name, phi3.5.

Going back to AOAI, with the new SDK when you request a ChatClient you provide the name of the AOAI deployment that you want to have the client for:

AzureOpenAIClient azureClient = new(
    new Uri("https://your-azure-openai-resource.com"),
    new DefaultAzureCredential());
ChatClient chatClient = azureClient.GetChatClient("my-gpt-4o-mini-deployment");

Via our docs. This is also the case with MEAI:

IChatClient client =
    new AzureOpenAIClient(
        new Uri(Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT")), 
        new DefaultAzureCredential())
            .AsChatClient(modelId: "gpt-4o-mini");

Aspire + AOAI

Presently, the AddDeployment method takes a AzureOpenAIDeployment which isn't a Resource type (and thus not returned), but let's propose a new API like so:

var aoai = builder.AddAzureOpenAI("aoai");
IResourceBuilder<AzureOpenAIDeploymentResource> gpt4o = aoai.AddDeploymentResource("gpt-4o");

builder.AddProject<Projects.Example>("api").WithReference(gpt4o);

Maybe the AddDeploymentResource would take in the AzureOpenAIDeployment type so you could get full configuration, maybe there'd be some defaults, that would have to be fleshed out. But the main thing that the AzureOpenAIDeploymentResource would do is provide a connection string with the endpoint/auth (from the parent) + DeploymentName=<deployment name>.

Then within our client app we would do:

builder.AddAzureOpenAIChatClient("gpt-4o");

This would register a ChatClient rather than (or in addition to) the OpenAIClient, and since it's using the connection name gpt-4o it would know to set that as the deploymentName by looking at the connection string of the AzureOpenAIDeploymentResource.

We could even adapt this same pattern to support MEAI and GitHub models.

For deployments, this wouldn't have any impact (from what I can gather) since the underlying types we have to create the AOAI resouce + deployments would still be leveraged, it really just comes down to the DX of working with deployments in AOAI.

And with development, the pattern of using a connection string could still be applied:

var gpt4o = builder.ExecutionContext.IsPublishMode
    ? builder.AddAzureOpenAI("aoai").AddDeployment("gpt-4o");
    : builder.AddConnectionString("gpt-4o");

builder.AddProject<Projects.ExampleProject>()
       .WithReference(gpt4o);

Working with multiple deployments on the same AOAI resource would look a little more verbose though:

IResourceBuilder<IResourceWithConnectionString>? chat;
IResourceBuilder<IResourceWithConnectionString>? embedding;

if (builder.ExecutionContext.IsPublishMode) {
    var aoai = builder.AddAzureOpenAI("aoai");
    chat = aoai.AddDeploymentResource("gpt-4o");
    embedding = aoai.AddDeploymentResource("ada-002");
} else {
    chat = builder.AddConnectionString("chat");
    embedding = builder.AddConnectionString("embedding");
}

builder.AddProject<Projects.ExampleProject>()
       .WithReference(chat)
       .WithReference(embedding);