ghost
commented
2 years ago Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.
Issue Details
### Description We are POC'ing a basic microservice architecture that uses the template _dotnet create webapi_ project and just adds an IHostedService to do background processing. Kestrel is used in order to be able to answer health checks from Kubernetes, so this basic architectural idea seems to make sense. Our background work for this POC involves pretty heavy interaction with AWS resources (SQS, S3, STS), and we are encountering SocketExceptions that have been very difficult to pin down. The canon guidance to use a static HttpClient instance application-wide does not work well when using the AWS clients in Amazon's SDK packages. The various AWS client constructors, and the config objects you can hand in, do not accept an instance of HttpClient. Instead, you must either rely on their built-in caching of HttpClient instance(s) that are created internally, or you have to provide an implementation that derives from their [HttpClientFactory](https://github.com/aws/aws-sdk-net/blob/master/sdk/src/Core/Amazon.Runtime/Pipeline/HttpHandler/_netstandard/HttpRequestMessageFactory.cs#L38). This is fairly straight-forward if you use [IHttpClientFactory](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-6.0) in .Net. The Create() method override that you implement simply calls .Create() on an instance of IHttpClientFactory provided via DI. However, even when providing this HttpClientFactory to all instances of any AWS client that is instantiated anywhere and everywhere in this service, it does not solve the SocketException problem. I will provide the stack trace below, but the properties of the actual SocketException object are extremely unhelpful: `Unknown socket error; ErrorCode: -131074; SocketErrorCode: SocketError; NativeErrorCode: -131074` The twist here is that these errors have only been encountered on Linux/in containers. Extended running tests on Windows cannot reproduce the problem when tested under load. The other twist is that our health checks calls are producing `System.InvalidOperationException: Handle is already used by another Socket.` errors (they are logged as Warning level) in Kestrel's handling pipeline. It would appear that Kestrel, and the HttpWebRequest pool used underneath IHttpClientFactory are stomping on each other. Given that this architectural choice of using a WebApplication to host a long-lived service doing background processing seems the most ideal and straightforward way of designing basic microservices, we would like to get to the bottom of this issue and figure out a way to have Kestrel and IHttpClientFactory play nice. If I am mischaracterizing the issue and there is a different cause, again, we'd very much like to get to the bottom of it. The fact that the runtime is throwing a SocketException with an error code that isn't even in the spec for TCP errors (nice article [here](https://blog.jetbrains.com/dotnet/2020/04/27/socket-error-codes-depend-runtime-operating-system/)) is the reason I am bringing it to this group. The runtime is burping up what would appear to be the equivalent of the `default` case in a `switch` statement. Any help or guidance here is appreciated. ### Reproduction Steps The basic structure is a web api project created with _dotnet create webapi_ with an `IHostedService` doing background processing interacting heavily with AWS resources. I can provide application code privately, as needed, so as to not have to share proprietary/private logic and access keys publicly. ### Expected behavior Interaction with AWS resources from a web application should not cause SocketExceptions. ### Actual behavior We are encountering SocketExceptions that have error codes that aren't even part of the TCP specs: `Unknown socket error; ErrorCode: -131074; SocketErrorCode: SocketError; NativeErrorCode: -131074` Full stack trace: `System.Net.Http.HttpRequestException: Unknown socket error (sqs.us-west-2.amazonaws.com:443) ---> System.Net.Sockets.SocketException (0xFFFDFFFE): Unknown socket error at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Sockets.Socket.
| Author: | sliderhouserules |
|---|---|
| Assignees: | - |
| Labels: | `area-System.Net.Http`, `untriaged` |
| Milestone: | - |
wfurt
tmds
sliderhouserules
antonfirsov
karelz
MMOSimca
stephentoub
Description
We have an application based on template dotnet create webapi project template and just adds an IHostedService to do background processing. Kestrel is used in order to be able to answer health checks from Kubernetes, as well as feed new requests into the service to be queued. Our background processing logic involves pretty heavy interaction with AWS resources (SQS, S3, STS), and we are encountering SocketExceptions that have been very difficult to pin down.
The canon guidance to use a static HttpClient instance application-wide does not work well when using the AWS clients in Amazon's SDK packages. The various AWS client constructors, and the config objects you can hand in, do not accept an instance of HttpClient. Instead, you must either rely on their built-in caching of HttpClient instance(s) that are created internally, or you have to provide an implementation that derives from their HttpClientFactory. This is fairly straight-forward if you use IHttpClientFactory in .Net. The Create() method override that you implement simply calls .Create() on an instance of IHttpClientFactory provided via DI.
However, even when providing this HttpClientFactory to all instances of any AWS client that is instantiated anywhere and everywhere in this service, it does not solve the SocketException problem. I will provide the stack trace below, but the properties of the actual SocketException object are extremely unhelpful:
Unknown socket error; ErrorCode: -131074; SocketErrorCode: SocketError; NativeErrorCode: -131074The twist here is that these errors have only been encountered on Linux/in containers. Extended running tests on Windows cannot reproduce the problem when tested under load.
The other twist is that our Kestrel calls are producing
System.InvalidOperationException: Handle is already used by another Socket.errors (they are logged as Warning level) in Kestrel's handling pipeline. It would appear that Kestrel, and the HttpWebRequest pool used underneath IHttpClientFactory are stomping on each other. If I am mischaracterizing the issue and there is a different cause, please let me know.The fact that the runtime is throwing a SocketException with an error code that isn't even in the spec for TCP errors (nice article here) is the reason I am bringing it to this group. The runtime is burping up what would appear to be the equivalent of the
defaultcase in aswitchstatement.Any help or guidance here is appreciated.
Reproduction Steps
The basic structure is a Web API project created with dotnet create webapi with an
IHostedServicedoing background processing interacting heavily with AWS resources. I can provide application code privately, as needed, so as to not have to share proprietary/private logic and access keys publicly.Expected behavior
Interaction with AWS resources from a web application should not cause SocketExceptions.
Actual behavior
We are encountering SocketExceptions that have error codes that aren't even part of the TCP specs:
Unknown socket error; ErrorCode: -131074; SocketErrorCode: SocketError; NativeErrorCode: -131074Full stack trace:
Regression?
The application in its current form is a .Net 6 solution. In a prior version, it was .Net 5 and the SocketExceptions were very rare if not non-existent.
However, even in the .Net 5 solution Kestrel was throwing the
System.InvalidOperationException: Handle is already used by another Socket.warnings when handling basic health check calls.Known Workarounds
We have implemented heavy retry policies using Polly, which allows the application to move past the errors, but it quite often takes a near complete restart of the background processing logic to clear up the SocketExceptions and allow the application to restart a job and create new connections to the AWS resources being used.
Configuration
The errors only appear running in containers on Linux.
Other information
N/A