OTLP exporter HttpRequestException: An error occurred while sending the request.

[muhaook]

Bug Report

Symptom

Describe the bug OTLP exporter failed on .net framework 4.6.2 with errors:

[2022-12-21T08:52:20.3657122Z] [Error] EventSource=OpenTelemetry-Exporter-OpenTelemetryProtocol, Message=Exporter failed send data to collector to https://aaaac4wtzq6x4aaaaaaaaaabpy.xxxx endpoint. Data will not be sent. 
Exception: System.Net.Http.HttpRequestException: An error occurred while sending the request. 
---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. 
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. 
---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
   at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
   at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.BaseOtlpHttpExportClient`1.SendHttpRequest(HttpRequestMessage request, CancellationToken cancellationToken)
   at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.BaseOtlpHttpExportClient`1.SendExportRequest(TRequest request, CancellationToken cancellationToken) 

OTLP exporter works on .net 6 for the same upload endpoint. So I think the backend should be good. it failed for some of my .net framework 4.6.2 samples hosted on IIS (webform/mvc/wcf). it succeeded for one of my console sample on .net framework 4.6.2.

Runtime environment:

• OpenTelemetry Automatic Instrumentation version: 0.5.1-beta.2
• OS: Windows 10
• .NET version: .NET Framework 4.6.2

otel-dotnet-auto-native-w3wp-1792.log WaimaiUi-2-133160873315949951-1792.log

[pjanotti]

Since it repros under IIS it makes me wonder if it is related to domain-neutral, see this comment.

[muhaook]

I am not sure if it is a domain-neutral issue. I did not see any assembly loading error in logs.

After googling, I tried to enable TLS 1.2 explicitly during initialization of auto-instrumentation. the issue seems fixed. But I am not sure if there are any side effects. Also, a question is whether we support TLS 1.2 in current auto-instrumentation, do we really need this fix?

+#if NETFRAMEWORK
+        string enableTls12 = System.Environment.GetEnvironmentVariable("ENABLE_TLS12");
+        if (string.IsNullOrEmpty(enableTls12) || enableTls12.Equals("true", StringComparison.OrdinalIgnoreCase))
+        {
+            System.Net.ServicePointManager.SecurityProtocol |= System.Net.SecurityProtocolType.Tls12;
+        }
+#endif

[pellared]

I suggest checking if it cannot be configured on OS level. See: https://learn.microsoft.com/en-us/dotnet/framework/network-programming/tls

Found here: https://learn.microsoft.com/pl-pl/dotnet/fundamentals/code-analysis/quality-rules/ca5386

[muhaook]

It makes sense, thanks. I experimented a little bit:

• added log information during auto-instrumentation initialization:

      Logger.Information("System.Net.ServicePointManager.SecurityProtocol: " + System.Net.ServicePointManager.SecurityProtocol.ToString());

• found a log entry in my sample on .Net framework 4.6.2 which showed that TLS 1.2 was not enabled:

  [2023-01-03T11:11:19.8575036Z] [Information] System.Net.ServicePointManager.SecurityProtocol: Ssl3, Tls 

• made change to my sampe by adding below line to web.config:

  <httpRuntime targetFramework="4.6.2" />

• then, I found a log entry as below which means TLS 1.2 was enabled

  [2023-01-03T11:18:08.9511968Z] [Information] System.Net.ServicePointManager.SecurityProtocol: Tls, Tls11, Tls12, Tls13 

• on a .Net 6 sample, I found a log entry below which means TLS depended on system:

  [2023-01-03T11:22:28.8355829Z] [Information] System.Net.ServicePointManager.SecurityProtocol: SystemDefault 

  I feel it is very helpful to have a log information to show current TLS support in dotnet-auto-instrumentation. I added it in Intiailize method; but it can be added to anywhere during dotnet-auto-instrumention initialization.

      Logger.Information("System.Net.ServicePointManager.SecurityProtocol: " + System.Net.ServicePointManager.SecurityProtocol.ToString());