grafana / faro-web-sdk

The Grafana Faro Web SDK, part of the Grafana Faro project, is a highly configurable web SDK for real user monitoring (RUM) that instruments browser frontend applications to capture observability signals. Frontend telemetry can then be correlated with backend and infrastructure data for full-stack observability.
https://grafana.com/oss/faro/
Apache License 2.0
749 stars 65 forks source link

Frontend Traces Linking To Backend Traces #705

Open tadkarshirish opened 5 days ago

tadkarshirish commented 5 days ago

## Description

We have enabled frontend observability for our Angular application using the Grafana Faro Web SDK and Faro Web Tracing CDN. The frontend traces are being sent to Grafana Tempo via Grafana Alloy, and they are visible in Tempo.

For our Java backend services, we are using the OpenTelemetry Operator with auto-instrumentation. Both the frontend and backend traces are being generated successfully but are appearing separately in Grafana Tempo.

According to the documentation, trace propagation should automatically link the frontend and backend traces, but this linkage is not happening in Grafana Tempo. We have confirmed that the traceparent header is present in the backend API request and matches the frontend trace. Despite this, no trace linkage is observed between the frontend and backend traces.

## Steps to reproduce

Versions:

Grafana Faro Web SDK : 1.10.2 Faro Web Tracing CDN : 1.10.2 OpenTelemetry Operator Helm Chart : v0.94.0 Java Auto-instrumentation Agent : 1.32.1

Frontend Setup: We integrated the Grafana Faro Web SDK and Faro Web Tracing CDN into the index.html of our Angular application. The app initiates a GET API call to a backend service for data.

The Angular app sends traces correctly, and they are visible in Grafana Tempo. Below is the index.html setup used to initialize Faro:

<script>
  window.addEventListener('environmentLoaded', (event) => {
    const { app_name, app_namespace, app_env } = event.detail;

    window.faro = window.GrafanaFaroWebSdk.initializeFaro({
      url: 'https://tempo.com/collect', 
      apiKey: 'your-api-key',
      app: {
        name: app_name,
        namespace: app_namespace,
        environment: app_env,
        version: '1.0.0',
      },
      instrumentations: [...window.GrafanaFaroWebSdk.getWebInstrumentations()],
    });

    window.addTracing();
  });

  window.addTracing = () => {
    if (window.GrafanaFaroWebSdk && window.GrafanaFaroWebTracing) {
      const tracingInstrumentation = new window.GrafanaFaroWebTracing.TracingInstrumentation();
      window.faro.instrumentations.add(tracingInstrumentation);

      const { trace, context } = window.faro.api.getOTEL();
      const tracer = trace.getTracer('default');
      const span = tracer.startSpan('initialization');

      context.with(trace.setSpan(context.active(), span), () => {
        span.setAttribute('page_url', window.location.pathname);
        span.end();
      });
    }
  };
</script>

Observations: When a frontend API call is made, we can see the traceparent header in the request headers sent to the backend service, and it matches the frontend trace:

traceparent: 00-74jfk49nslkc0485b0859djsc5-3ldskskxcc-01 this trace id send request headers to backend is same as frontend generate trace. but backend service generating new traceid for this request. the traces do not link in Grafana Tempo.

Backend Setup: We are using OpenTelemetry Operator with the following configuration for auto-instrumenting our Java backend services:

yaml

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: java-instrumentation
spec:
  exporter:
    endpoint: "http://centralcollector.tracing.svc.cluster.local:4317"
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: always_on
  java: 
    image: "imagepath/tracing/autoinstrumentation/java:1.32.1"
    resources:
      limits:
        cpu: 500m
        memory: 450Mi
      requests:
        cpu: 150m
        memory: 230Mi   
  env:
    - name: OTEL_METRICS_EXPORTER
      value: none
    - name: OTEL_LOG_LEVEL
      value: debug

## Expected behavior We expect Grafana Tempo to show the complete trace flow, from the frontend request in the Angular application to the backend response in the Java service. The traces from the frontend and backend should be automatically linked using trace propagation.

undefinedhuman commented 5 days ago

Hi there, maybe it helps; for me the problem was my Backend did not attach the "server-timing" header back to the response:

Grafana Cloud Docs

I solved this via the recommended approach of integrating the middleware in my backend: https://grafana.com/docs/grafana-cloud/monitor-applications/frontend-observability/apm-integration/#nodejs-example

Statement from the docs:

After the server sends a server-timing header, the RUM instrumentation automatically picks it up. From the user session view, you can navigate directly to the trace that generated the response. In this example, the initial navigation links to the request loading the page. Click the Services action to the right of the row to continue the investigation in the Application Observability space.