OTLP exporter Exception: System.Net.Http.HttpRequestException: Error while copying content to a stream.

[muhaook]

Bug Report

Symptom

hit a bunch of OTLP exporter exceptions during load-test:

[2023-02-21T09:52:49.5020419Z] [Error] EventSource=OpenTelemetry-Exporter-OpenTelemetryProtocol, Message=Exporter failed send data to collector to https://<hostname>:5679/<path> endpoint. Data will not be sent. 

Exception: System.Net.Http.HttpRequestException: Error while copying content to a stream. 
---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. 
---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine

   at System.Net.Sockets.Socket.BeginSend(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
   at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at System.Net.Security._SslStream.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncCallback asyncCallback, Object asyncState)
   at System.Net.TlsStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback asyncCallback, Object asyncState)
   at System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at System.Net.ConnectStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at System.IO.Stream.<>c.<BeginEndWriteAsync>b__53_0(Stream stream, ReadWriteParameters args, AsyncCallback callback, Object state)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrim[TInstance,TArgs](TInstance thisRef, TArgs args, Func`5 beginMethod, Func`3 endMethod)
   at System.IO.Stream.BeginEndWriteAsync(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.Stream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.BaseOtlpHttpExportClient`1.SendHttpRequest(HttpRequestMessage request, CancellationToken cancellationToken)
   at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.BaseOtlpHttpExportClient`1.SendExportRequest(TRequest request, CancellationToken cancellationToken) 

Runtime environment (please complete the following information):

• OpenTelemetry Automatic Instrumentation version: v0.5.1-beta.2
• OS: Windows Server 2012r2
• .NET version: .NET Framework 4.6.2

[pellared]

Thanks a lot for reporting the issue 👍

OpenTelemetry Automatic Instrumentation version: v0.5.1-beta.2 uses https://www.nuget.org/packages/OpenTelemetry.Exporter.OpenTelemetryProtocol/1.4.0-rc.2

Can you please try using https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/tag/v0.6.0-beta.2 which uses https://www.nuget.org/packages/OpenTelemetry.Exporter.OpenTelemetryProtocol/1.4.0-rc.4 ?

If you will be able to reproduce it then we will probably have to escalate it to @open-telemetry/dotnet-maintainers becasue AFAIK they plan to make a 1.4.0 by end of this week.

It would be great to have something some artifacts that would help to reproduce the issue, but first it would be great if you just retest with the latest OTel .NET Auto-Instrumentation version.

[muhaook]

Thanks @pellared , we are targeting v0.5.1-beta.2 at the moment. load test is running by another team. so it may not be straight forward to upgrade. I'll see if we can switch to v0.6.0-beta.2 as soon as possible.

for info, I did a quick search and found it may be the other side that aborted the connection. https://stackoverflow.com/questions/14304658/c-sharp-an-established-connection-was-aborted-by-the-software-in-your-host-machi

[pellared]

I think that according to https://github.com/open-telemetry/opentelemetry-specification/blob/v1.18.0/specification/protocol/exporter.md#retry the such HTTP request SHOULD be retried, but I cannot find such code in https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/src/OpenTelemetry.Exporter.OpenTelemetryProtocol

Moving to https://github.com/open-telemetry/opentelemetry-dotnet (feel free to move it back to https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation if necessary) to get more 👀 on this issue

[Kielek]

Ref link: #4115

[github-actions[bot]]

This issue was marked stale due to lack of activity and will be closed in 7 days. Commenting will instruct the bot to automatically remove the label. This bot runs once per day.