How to add OpenTelemetry for .NET framework applications hosted in IIS?

[nakbsaxo]

I am trying to add OpenTelemetry for .NET framework applications hosted in IIS using https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation?tab=readme-ov-file documentation while checking the log files I can see lots of similar errors.

Code:

$appPools = Get-IISAppPool
$appPoolNames = $appPools.Name
$appPoolNames.Count

foreach($appPoolName in $appPoolNames)
{
    if($appPoolName -ne "DefaultAppPool")
    {
        $appcmd = "$($env:systemroot)\system32\inetsrv\AppCmd.exe"
        $appPool = $appPoolName

        $environment = @{
            OTEL_RESOURCE_ATTRIBUTES="name=$appPool,service.version=1.0.0"
            OTEL_SERVICE_NAME=$appPool
            COR_ENABLE_PROFILING=1
            COR_PROFILER="{918728DD-259F-4A6A-AC2B-B85E1B658318}"
            COR_PROFILER_PATH_64="C:\Program Files\opentelemetry-dotnet-instrumentation-windows\win-x64\OpenTelemetry.AutoInstrumentation.Native.dll"
            OTEL_DOTNET_AUTO_HOME="C:\Program Files\opentelemetry-dotnet-instrumentation-windows"
            OTEL_EXPORTER_OTLP_ENDPOINT=<URL>
            OTEL_DOTNET_AUTO_TRACES_CONSOLE_EXPORTER_ENABLED="true"
            OTEL_DOTNET_AUTO_METRICS_CONSOLE_EXPORTER_ENABLED="true"
            OTEL_DOTNET_AUTO_LOGS_CONSOLE_EXPORTER_ENABLED="true"
        }

        $environment.Keys | ForEach-Object {
          & $appcmd set config -section:system.applicationHost/applicationPools /+"[name='$appPool'].environmentVariables.[name='$_',value='$($environment[$_])']"
        }
    }
}

Error:

EventSource=OpenTelemetry-AutoInstrumentation, Message=The profiler has been initialized with 0 derived definitions. 
EventSource=OpenTelemetry-Api, Message=Failed to inject activity context in format: 'TraceContextPropagator', context: 'Invalid context'.

[Kielek]

@nakbsaxo, I think that you have reached this line https://github.com/open-telemetry/opentelemetry-dotnet/blob/b9d56aa64ecbad7a56c61941a760abd210051590/src/OpenTelemetry.Api/Context/Propagation/TraceContextPropagator.cs#L100

Could you please provide us Minimal, Reproducible Example? With exact calls to the endpoint. It will be easier to give you advice. I suppose that you really do not have any activity context and OTel is to strict to log warning.

[nakbsaxo]

Hi @Kielek thanks, I am setting above environment variable in each app pool to inject OpenTelemetry libraries and seeing the above warning in logs not sure why not all the traces it capture by the way it's dev environment where I am testing where traffic not too much but still having decent traffic.

Could you pleaase let me know what exactly do I need to share.

[nakbsaxo]

Hi @Kielek I can also see below error though I am using OTEL_EXPORTER_OTLP_ENDPOINT

EventSource=OpenTelemetry-Exporter-OpenTelemetryProtocol, Message=Exporter failed send data to collector to http://localhost:4318/v1/traces endpoint. Data will not be sent. Exception: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:4318 at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult) at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception) --- End of inner exception stack trace --- at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context) at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar) --- End of inner exception stack trace --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.BaseOtlpHttpExportClient1.SendHttpRequest(HttpRequestMessage request, CancellationToken cancellationToken) at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.BaseOtlpHttpExportClient1.SendExportRequest(TRequest request, DateTime deadlineUtc, CancellationToken cancellationToken) [2024-05-30T10:08:36.1144906Z] [Warning] EventSource=OpenTelemetry-Api, Message=Failed to inject activity context in format: 'TraceContextPropagator', context: 'Invalid context'. [2024-05-30T10:08:38.1266251Z] [Error] EventSource=OpenTelemetry-Exporter-OpenTelemetryProtocol, Message=Exporter failed send data to collector to http://localhost:4318/v1/traces endpoint. Data will not be sent. Exception: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:4318 at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult) at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception) --- End of inner exception stack trace --- at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context) at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar) --- End of inner exception stack trace --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.BaseOtlpHttpExportClient1.SendHttpRequest(HttpRequestMessage request, CancellationToken cancellationToken) at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.BaseOtlpHttpExportClient1.SendExportRequest(TRequest request, DateTime deadlineUtc, CancellationToken cancellationToken)

[Kielek]

@nakbsaxo, sorry for the late response, I have missed your comment.

Exporter failed send data to collector to http://localhost:4318/v1/traces endpoint. Data will not be sent. Exception: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:4318

This clearly means that the exporter is not able to send data to your collector. In this case, you are trying to sent to default OTLP over http/protobuf. The collector (endpoint) should be on your local machine.

Is it what you are trying to achieve? What is the value of OTEL_EXPORTER_OTLP_ENDPOINT visible for the process?

[nakbsaxo]

@Kielek I have given the exporter url which is http://otc-gw-collector-lb..xxx:4318 which is working fine now but getting below warning could you please let me know what does that mean or any impact on the traces.

EventSource=OpenTelemetry-AutoInstrumentation, Message=The profiler has been initialized with 0 derived definitions. 
EventSource=OpenTelemetry-Api, Message=Failed to inject activity context in format: 'TraceContextPropagator', context: 'Invalid context'.

[Kielek]

EventSource=OpenTelemetry-AutoInstrumentation, Message=The profiler has been initialized with 0 derived definitions. 

It is not a Warning, it is information level. It is perfectly normal, standard behavior. For now, there is no bytecode instrumentation for derived types in this repository. It is feature which can be needed and occurs in our code due to donation made by DataDog. See https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/d2a7170fb4fd85ac6656590f6fd52277bd0d8701/src/OpenTelemetry.AutoInstrumentation/Instrumentation.cs#L202

EventSource=OpenTelemetry-Api, Message=Failed to inject activity context in format: 'TraceContextPropagator', context: 'Invalid context'.

As you know, it seems to be related to https://github.com/open-telemetry/opentelemetry-dotnet/issues/4247 I am not sure if anybody is actively working on this.

[nakbsaxo]

Hi @Kielek I am getting some data not all the traces and some apppool is also crashing and I can see below error in application logs any ide?

[Kielek]

@nakbsaxo, it looks like an issue described in https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/549c36a4f7f62ec0c3245d470518b86f96ac253e/docs/troubleshooting.md#assembly-version-conflicts

If it does not help you, please provide Minimal, Reproducible Example so we could debug this issue.

[Kielek]

Closing as stale. Feel free to reopen when needed.