Automate seamless correlation with Grafana Frontend Observability

[cedricziel]

Feature Request

Grafana Frontend Observability can correlate individual page - loads and asynchronous backend calls automatically with Application Observability. The RUM and APM correlation need is pretty popular to diagnose slow loading sites and generally correlate across product areas.

However creating strong correlation, the server needs to send a response header of server-timing, containing the traceparent field of the root span. Implementing this in every service is very expensive. We should look for a solution that decreases the time to value for users and automatically inject the current traceparent for http responses.

Describe the solution you'd like:

Inject a server-timing header containing the current traceparent in its description so that the Faro Web SDK can capture it from the browser performance API.

grafana/faro-web-sdk@ce86005/packages/web-sdk/src/instrumentations/performance/navigation.test.ts#L96-L109

Describe alternatives you've considered.

Which alternative solutions or features have you considered?

Additional Context

Sprungs OTel distribution does it here https://github.com/signalfx/splunk-otel-dotnet/blob/main/src/Splunk.OpenTelemetry.AutoInstrumentation/Traces.cs

Relevant change in Java OpenTelemetry https://github.com/grafana/grafana-opentelemetry-java/pull/661

[matt-hensley]

Unfortunately the existing instrumentation only allows for registration of a single delegate when doing response enrichment. It's possible to combine delegates in .NET, but getting access to the the multiple delegates to combine them isn't straightforward.

The .NET auto-instrumentation makes this relatively straightforward due to it's plugin infrastructure. Similarly, the Java agent has an extension mechanism to customize behaviors.

The idiomatic way of configuring the instrumentation is using the built-in configuration API. In the following pseudo code, only the configuration registered in the application would run, as last one registered wins:

// inside the distro
services.Configure<AspNetCoreTraceInstrumentationOptions>(options =>
{
    options.EnrichWithHttpResponse = (activity, httpResponse) =>
    {
        // add header to httpResponse
    };
});

// inside application code
services.Configure<AspNetCoreTraceInstrumentationOptions>(options =>
{
    options.EnrichWithHttpResponse = (activity, httpResponse) =>
    {
        // this overrides previous EnrichWithHttpResponse delegate
    };
});

I've raised this issue upstream, and it's been brought up with the new developer experience SIG.