Lack of Traces when Using Elastic APM RUM with Frontend Instrumentation

b2ronn commented 1 year ago

I'm using Tracetest with Elastic APM. All applications are instrumented with Elastic agents. Upon integrating Elastic APM with the frontend part of the application using the instructions provided https://www.elastic.co/guide/en/apm/agent/rum-js/current/install-the-agent.html#_synchronous_blocking_pattern, I'm unable to capture traces when executing an HTTP Request test in Tracetest. Instead, the Tracetest web interface doesn't display any traces. Additionally, the trace does not appear in the Kibana.

Steps to Reproduce:

Create a page with the integrated Elastic APM RUM agent using the following code:

<!DOCTYPE html>
<html>
<head>
    <title>Elastic APM RUM Example</title>
</head>
<body>
    <h1>Now</h1>
    <p id="demo"></p>
    <script src="https://unpkg.com/@elastic/apm-rum@5.12.0/dist/bundles/elastic-apm-rum.umd.min.js"></script>
    <script>
        const apm = elasticApm.init({
            serviceName: 'some-name',
            serverUrl: 'https://apmserver.com',
            serviceVersion: '0.5.11'
        });
        apm.addLabels({ project: 'some-labels' });
    </script>
    <script>
        const d = new Date();
        document.getElementById("demo").innerHTML = d;
    </script>
</body>
</html>

Launch the Tracetest web interface and perform an HTTP Request test on the created page.

Expected Behavior:

It is expected that the Tracetest interface will display traces associated with the execution of the HTTP Request test on the page with the integrated Elastic APM RUM agent.

Actual Behavior:

The Tracetest web interface does not show any traces after performing the HTTP Request test. There is no information about traces related to the integrated Elastic APM RUM agent. Additionally, the trace does not appear in the Kibana.

Additional Information:

All applications are instrumented with Elastic agents. The issue is observed only when integrating with the frontend part and performing an HTTP Request test in Tracetest.

mathnogueira commented 1 year ago

@b2ronn I have a couple of questions:

When you run the test, can you still see your trace in the APM?
Can you check if the trace-id matches the only from Tracetest? You can see the expected trace-id from tracetest in the information ballon on the right of the test name

If the trace-id from Tracetest doesn't match the trace-id in the trace in your APM, it means your app is ignoring the trace propagation headers in the original request, this might be the root cause of the issue.

b2ronn commented 1 year ago

I do not see trace in Kibana when I run a test from Tracetest. Enabled debug on the APM server shows that nothing is being received. If the page is opened in a browser, the trace appears.

little incorrect. i only see one trace (tracetest trigger) that tracetest sends through the OpenTelemetry collector, but there are no frontend spans.

mathnogueira commented 1 year ago

I think I understood the issue. Elastic APM RUM only tracks real users. One of the first filters they have to decide if a user is real or not is to identify if the request comes from a browser. As Tracetest is sending an HTTP request to the server, this condition fails and the APM RUM agent doesn't start. That's the reason you get no traces when you run your app with Tracetest. You will probably face the same issue if you use Postman to trigger your app.

That condition makes sense due to the kind of information that the agent tries to collect from your webpage. In that case, I don't think there's an easy workaround for that issue.

I'd suggest you use OpenTelemetry's JS library to instrument your frontend app instead of RUM. RUM is awesome for monitoring real usage but seems suboptimal for tracing. You can check the docs about browser instrumentation here. Although it's experimental, we have users using it and being successful, We also have built some PoCs with it in the past and it worked fine.

b2ronn commented 1 year ago

It seems that because Go-http-client doesn't process JS scripts on the page, traces are also not sent.

mathnogueira commented 1 year ago

You are right, it will only fetch the request body as text and not run it. To overcome this, we would probably need to support triggering something with a headless browser. I think such a feature would be extremely useful for end-to-end tests. I'll add it to our backlog, but I don't think we have the throughput to work on it in the near future. But Pull requests from the community are more than welcome :smile:

kubeshop / tracetest

Lack of Traces when Using Elastic APM RUM with Frontend Instrumentation #3104