.Net: Bug: .NET Bedrock streaming completion is keeping the request thread busy

tlecomte commented 1 week ago

Describe the bug The current implementation of the Bedrock connector is using response.Stream.AsEnumerable() in BedrockChatCompletionClient.cs#L193, which is keeping the current .NET thread busy while it reads the response stream. This could cause thread exhaustion issues.

To Reproduce Steps to reproduce the behavior:

See code at https://github.com/microsoft/semantic-kernel/blob/f88cf5f147f9020a2607fee45f17025fac864db0/dotnet/src/Connectors/Connectors.Amazon/Bedrock/Core/Clients/BedrockChatCompletionClient.cs#L193

Expected behavior Replace .AsEnumerable() with something that would read the response stream in an async manner.

Screenshots N/A

Platform

OS: all
IDE: N/A
Language: C#
Source: NuGet package version 1.26.0

Additional context

tlecomte commented 1 week ago

I realize that, ideally, the AWS .NET SDK would provide a solution out of the box. So I've also created https://github.com/aws/aws-sdk-net/issues/3542 and will suggest a solution there first.

evchaki commented 1 week ago

@RogerBarreto - can you take a look at this? @tlecomte - thanks for submitting this, we will take a look at updating this.

microsoft / semantic-kernel

.Net: Bug: .NET Bedrock streaming completion is keeping the request thread busy #9519

Additional context