Out-of-Process Auto-Instrumentation for .NET AOT Applications

[rajkumar-rangaraj]

Problem Statement

Auto-instrumentation for .NET applications currently relies on startup hooks, which are not supported by Ahead-of-Time (AOT) compiled applications. This lack of support for AOT-compiled apps is a significant gap in OpenTelemetry .NET Auto-Instrumentation capabilities. Additionally, some apps may fail rule engine validation and also require alternative methods for auto-instrumentation.

Goals

• Extend OpenTelemetry .NET auto-instrumentation to support AOT compiled apps
• Provide a mechanism for apps that fail rule engine validation to be auto-instrumented

Proposed Solution

The central idea is to employ an out-of-process auto-instrumentation technique that leverages EventPipeSession. This will be similar in function to utilities like dotnet-counter and dotnet-trace.

// Setup EventPipeSession
var client = new DiagnosticsClient(processId);
using (EventPipeSession session = client.StartEventPipeSession(providers, requestRundown: false))
{
  var source = new EventPipeEventSource(session.EventStream);
  Task.Run(() =>
  {
      source.Dynamic.All += (TraceEvent data) => 
      {
        Console.WriteLine($"Event: {data.EventName}");
      };
    source.Process();
  });

Compatibility

This feature would be opt-in and should not break existing auto-instrumentation methods. The out-of-process method would serve as an alternative to startup hooks for AOT apps and others where in-process methods are not feasible.

Test Plan

• Unit Tests: Write unit tests to verify the EventPipeSession setup and data collection.
• Integration Tests: Check if existing tests can be used for the new AOT-compatible method.

Rollout Plan

• Alpha / Beta Release: Introduce the feature as an alpha/beta release and collect feedback from customers.
• Full Release: After sufficient testing and community feedback, roll out the feature in a full release.

Open Questions

1. Are there limitations in EventPipe that could make this method not universally applicable?
2. What is the performance overhead in real-world scenarios?

[nrcventura]

Other thoughts/considerations:

• Should an exporter plugin be provided for an existing tool like dotnet-monitor in order to transmit the data using OTLP?
• How compliant to the OTel semantic conventions will the data be?
  • If the data is not compliant, is there enough information available to transform it into compliance?
• Do we need to put some protections in place to ensure that the event pipe is not used "in-process"? I've seen some instability with the application process when subscribing to it from an "in-process" context, leading to an affect on the exit code of the running application (caused by a memoryaccessviolation). This could also help with preventing double "instrumentation".

[pellared]

What about context propagation? Would it be able to work?

Maybe it would be possible to build automatic instrumentation using eBPF similar to OpenTelemetry Go Automatic Instrumentation: https://github.com/open-telemetry/opentelemetry-go-instrumentation/blob/main/docs/how-it-works.md

[nrcventura]

Closing per SIG discussion. Pursuing dotnet-monitor based approach.