Open lucasmeijer opened 1 week ago
Needs review with the team.
The normal request / response APIs are using async don't the http client but I agree after revisiting the for response streaming that the StartProcessing
method does eventually go down to a loop calling Stream.Read
on the stream return back for the response and that will block till there is data coming back from Bedrock.
At a minimum we should rework the ProcessLoop to use async / await.
I created a PR to add a new StartProcessingAsync
method that will use async methods down to the network stream read. https://github.com/aws/aws-sdk-net/pull/3364
Describe the bug
This is unfortunate. The threads that are supposed to handle new incoming requests for my server, are now blocked on very long running blocking network calls. (AI model responses are easily 30s, (spread out over many events)).
Expected Behavior
I would expect a way that allows me to use async enumeration:
EnumerateEventsAsync would under the hood not use blocking io, leaving my servers threads available to do server work, and come back to this network request only when data is there.
Current Behavior
server threads being blocked.
Reproduction Steps
see code snippet in description
Possible Solution
No response
Additional Information/Context
No response
AWS .NET SDK and/or Package version used
Targeted .NET Platform
net8
Operating System and version
all