[API Proposal]: System.ClientModel service method collection return types

[annelo-msft]

Background and motivation

Overview

The System.ClientModel library provides building blocks for .NET clients that call cloud services. System.ClientModel 1.0.0 has types for reading and writing message content, reviewed as part of https://github.com/dotnet/runtime/issues/94126, and System.ClientModel 1.1.0-beta.2 added end-user convenience types and client-author pipeline types, reviewed as part of https://github.com/dotnet/runtime/issues/97711.

In this iteration of the System.ClientModel API, we would like to add types to enable clients to provide service methods to retrieve collections of items from paginated endpoints and streaming endpoints.

Page Collection Types

Cloud services use pagination to return a collection of items over multiple responses. Each response from the service returns a page of items in the collection, as well as the information needed to obtain the next page of items, until all the items in the requested collection have been returned. There are two primary client-user scenarios for this category of service methods: enumerating all the items of the requested collection (independent of the paged delivery mechanism) and rendering a single page of results. In addition to these primary scenarios, there is a requirement that the collection must be able to be "rehydrated," i.e. continued from a process different from the one that originally created the collection. The rehydration requirement means that we must be able to persist the collection state and separately recreate the collection from the persisted state.

To enable clients to provide service methods for paginated endpoints, we would like to add PageCollection<T>, AsyncPageCollection<T>, PageResult<T> and ContinuationToken types as described below.

Streaming Collection Types

Cloud services such as OpenAI and Azure OpenAI use SSE streams to deliver incremental updates sequentially over a single response stream. We would like to add CollectionResult<T> and AsyncCollectionResult<T> types to System.ClientModel to provide a convenience layer over the SseParser types in the System.Formats.Sse namespace. This will enable clients to provide service methods that return collections of model types and from which the raw HTTP response details can also be obtained.

API Proposal

namespace System.ClientModel
{
    public abstract partial class AsyncCollectionResult<T> : System.ClientModel.ClientResult, System.Collections.Generic.IAsyncEnumerable<T>
    {
        protected internal AsyncCollectionResult() { }
        protected internal AsyncCollectionResult(System.ClientModel.Primitives.PipelineResponse response) { }
        public abstract System.Collections.Generic.IAsyncEnumerator<T> GetAsyncEnumerator(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken));
    }
    public abstract partial class AsyncPageCollection<T> : System.Collections.Generic.IAsyncEnumerable<System.ClientModel.PageResult<T>>
    {
        protected AsyncPageCollection() { }
        public System.Collections.Generic.IAsyncEnumerable<T> GetAllValuesAsync([System.Runtime.CompilerServices.EnumeratorCancellationAttribute] System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
        protected abstract System.Collections.Generic.IAsyncEnumerator<System.ClientModel.PageResult<T>> GetAsyncEnumeratorCore(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken));
        public System.Threading.Tasks.Task<System.ClientModel.PageResult<T>> GetCurrentPageAsync() { throw null; }
        protected abstract System.Threading.Tasks.Task<System.ClientModel.PageResult<T>> GetCurrentPageAsyncCore();
        System.Collections.Generic.IAsyncEnumerator<System.ClientModel.PageResult<T>> System.Collections.Generic.IAsyncEnumerable<System.ClientModel.PageResult<T>>.GetAsyncEnumerator(System.Threading.CancellationToken cancellationToken) { throw null; }
    }
    public abstract partial class CollectionResult<T> : System.ClientModel.ClientResult, System.Collections.Generic.IEnumerable<T>, System.Collections.IEnumerable
    {
        protected internal CollectionResult() { }
        protected internal CollectionResult(System.ClientModel.Primitives.PipelineResponse response) { }
        public abstract System.Collections.Generic.IEnumerator<T> GetEnumerator();
        System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() { throw null; }
    }
    public partial class ContinuationToken
    {
        protected ContinuationToken() { }
        protected ContinuationToken(System.BinaryData bytes) { }
        public static System.ClientModel.ContinuationToken FromBytes(System.BinaryData bytes) { throw null; }
        public virtual System.BinaryData ToBytes() { throw null; }
    }
    public abstract partial class PageCollection<T> : System.Collections.Generic.IEnumerable<System.ClientModel.PageResult<T>>, System.Collections.IEnumerable
    {
        protected PageCollection() { }
        public System.Collections.Generic.IEnumerable<T> GetAllValues() { throw null; }
        public System.ClientModel.PageResult<T> GetCurrentPage() { throw null; }
        protected abstract System.ClientModel.PageResult<T> GetCurrentPageCore();
        protected abstract System.Collections.Generic.IEnumerator<System.ClientModel.PageResult<T>> GetEnumeratorCore();
        System.Collections.Generic.IEnumerator<System.ClientModel.PageResult<T>> System.Collections.Generic.IEnumerable<System.ClientModel.PageResult<T>>.GetEnumerator() { throw null; }
        System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() { throw null; }
    }
    public partial class PageResult<T> : System.ClientModel.ClientResult
    {
        internal PageResult() { }
        public System.ClientModel.ContinuationToken? NextPageToken { get { throw null; } }
        public System.ClientModel.ContinuationToken PageToken { get { throw null; } }
        public System.Collections.Generic.IReadOnlyList<T> Values { get { throw null; } }
        public static System.ClientModel.PageResult<T> Create(System.Collections.Generic.IReadOnlyList<T> values, System.ClientModel.ContinuationToken pageToken, System.ClientModel.ContinuationToken? nextPageToken, System.ClientModel.Primitives.PipelineResponse response) { throw null; }
    }
}

API Usage

Example usage of AsyncCollectionResult<T> with OpenAI chat completions operation:

AsyncCollectionResult<StreamingChatCompletionUpdate> updates
    = client.CompleteChatStreamingAsync("Say 'this is a test.'");

Console.WriteLine($"[ASSISTANT]:");
await foreach (StreamingChatCompletionUpdate update in updates)
{
    foreach (ChatMessageContentPart updatePart in update.ContentUpdate)
    {
        Console.Write(updatePart.Text);
    }
}

Example usage of `AsyncPageCollection with OpenAI list assistants operation:

AsyncPageCollection<Assistant> assitantPages = client.GetAssistantsAsync();
IAsyncEnumerable<Assistant> assistants = assitantPages.GetAllValuesAsync();
await foreach (Assistant assistant in assistants)
{
    Console.WriteLine($"[{count,3}] {assistant.Id} {assistant.CreatedAt:s} {assistant.Name}");

    count++;
}

Sample usage of PageResult<T> in ASP.NET web app "collection rehydration" scenario:

PageCollection<Assistant> assistantPages = (cachedPageTokenBytes is null) ?

    // We don't have a token cached for the page of results to render.
    // Request a new collection from user inputs.
    assistantClient.GetAssistants(new AssistantCollectionOptions()
    {
        Order = listOrder,
        PageSize = pageSize
    }) :

    // We have a serialized page token that was cached when a prior
    // web app page was rendered.
    // Rehydrate the page collection from the cached page token.
    assistantClient.GetAssistants(ContinuationToken.FromBytes(cachedPageTokenBytes));

// Get the current page from the collection.
PageResult<Assistant> assitantPage = assistantPages.GetCurrentPage();

// Set the values for the web app page to render.
Assistants = assitantPage.Values;

// Cache the serialized page token value to use the next time
// the web app page is rendered.
CacheBytes("PageToken", assitantPage.PageToken.ToBytes());

A full implementation of a mock paging client can be found here.

Alternative Designs

No response

Risks

No response

[dotnet-policy-service[bot]]

Tagging subscribers to this area: @dotnet/area-system-collections See info in area-owners.md if you want to be subscribed.

[julealgon]

Is there any expectation to ever use these new base system types to also represent paged data from OData services?

@xuzhg @habbes

[annelo-msft]

Is there any expectation to ever use these new base system types to also represent paged data from OData services?

These types are intended for use with System.ClientModel-based clients, and such clients can talk to cloud services that use OData in their APIs. I don't know that there is an expectation that they would be used outside the SCM-based client context, but if they are useful outside of that context, I don't know of reasons not to use them. One limitation might be that they are currently scoped to HTTP given the dependency on SCM's PipelineResponse.

[eiriktsarpalis]

The additions would still live in the Azure SDK repo and the NuGet package correct?

[annelo-msft]

The additions would still live in the Azure SDK repo and the NuGet package correct?

That is correct. The System.ClientModel code is here and the NuGet package is here.

[eiriktsarpalis]

I set the milestone to 10.0.0 since we're currently in the process on bringing down our issue numbers for .NET 9 endgame. Are you looking to get this reviewed soon?

[stephentoub]

Can you help me better understand ContinuationToken? What does someone do with the bytes when handed an arbitrary token? Seems like a given token instance would only make sense to the API set that produced it? Do the bytes-related APIs even need to be part of the base type? Or are these bytes the rehydration mechanism you referred to, such that they're exposed purely so they can be persisted and then used to reconstitute a token? And if that's the case, the ctor that takes bytes doesn't need to be public?

[annelo-msft]

Can you help me better understand ContinuationToken?

Yes, good questions, and glad to get your eyes on this area.

This type was designed to satisfy the following constraints:

1. Can hold all the information needed to recreate a page collection at its current index from a different process than the one that originally obtained the collection instance
2. Can be persisted and created from byte[] to work well with ASP.NET IDistributedCache APIs
3. Won't require adding many operation-specific types to clients' public API surface

To satisfy 3, the intended use for this type is that clients will create internal subtypes of ContinuationToken specific to a service method for a paginated service endpoint. An example is MessagesPageToken. It can create itself from either bytes or a runtime instance when passed to a client method, for example GetMessages.

I looked at using only BinaryData instead of adding a new ContinuationToken type, but this would have required always writing the bytes of a token to create a PageResult<T> instance, even though paying that perf cost would be needed in only a small percentage of use cases.

What does someone do with the bytes when handed an arbitrary token?

I put a snip from a sample ASP.NET app above, but the primary use case is to be able to resume paging from processes behind a load balancer. One process can cache a byte[] from a page token and another can retrieve the byte[] and recreate the collection from its current state by doing something like:

assistantClient.GetAssistants(ContinuationToken.FromBytes(cachedPageTokenBytes));

Seems like a given token instance would only make sense to the API set that produced it?

That is correct. A client should throw if an instance is passed to a service method that can't be used to create the correct internal token type.

Do the bytes-related APIs even need to be part of the base type? Or are these bytes the rehydration mechanism you referred to, such that they're exposed purely so they can be persisted and then used to reconstitute a token?

Yes, the latter. These APIs allow us to defer writing the runtime type to bytes until a user opts-in to the "rehydration" scenario by calling ToBytes.

And if that's the case, the ctor that takes bytes doesn't need to be public?

I think it does, since the operation-specific types the client is using are internal.

(All of that said, if you see ways to improve these APIs, I am always open to making the product better!)

As a side note, we also plan to use ContinuationToken to implement rehydration scenarios for the SCM long-running operation type that is forthcoming.

Thanks!

[terrajobst]

That makes sense. Thanks for keeping us in the loop. Since we don't have any change requests, we'll close this as completed.