search

artilleryio / artillery

The complete load testing platform. Everything you need for production-grade load tests. Serverless & distributed. Load test with Playwright. Load test HTTP APIs, GraphQL, WebSocket, and more. Use any Node.js module.
https://www.artillery.io
Mozilla Public License 2.0
8.05k stars 512 forks source link

Datadog dashboard data missing #3342

Open AT2406 opened 2 months ago

AT2406 commented 2 months ago

Hello, we have implemented OTEL with our artillery playwright tests and it appears that some of our data is inconsistent on our dashboard.

For example, we have implemented a custom replace regex for span names on navigation to specific pages, but these number of navigations being recorded are not the same as what we see for page views in GA.

We have even estimated the page views based on successful transactions that trigger a page view and the numbers do not align (SEE ATTACHED IMAGE). We have checked our OTEL collector and data dog and there is no sample rate/limit happening. We wondered if there is potentially any way artillery could be skipping traces or if there is some sort of queuing system in place for sending traces? Also, does artillery only report hard-navigations? Or will it report soft navigations as well?

image (1)

hassy commented 2 months ago

hi @AT2406, thanks for creating the issue!

We wondered if there is potentially any way artillery could be skipping traces or if there is some sort of queuing system in place for sending traces?

There is an internal queue but our OTel integration implements some logic to make sure everything is sent out before Artillery exits here: https://github.com/artilleryio/artillery/blob/main/packages/artillery-plugin-publish-metrics/lib/open-telemetry/index.js#L208

Also, does artillery only report hard-navigations? Or will it report soft navigations as well?

Hard navigations only, we use on the web-vitals package internally to report Web Vitals and would need soft navigations to be supported there first.

AT2406 commented 2 months ago

Ah okay thank you for clarifying this for me, is there anyway to exclude specific traces? i.e. the Page: https://xyz/ ones?

sam-barker commented 2 months ago

Hey @hassy Thanks for getting back to @AT2406 in regards to this.

We're having quite big issues with missing spans and we're trying to work out why.

Screenshot 2024-09-19 at 14 17 49

This is a graph of a small test that I have ran, it shows the number of different page types that have been accessed as well as the page views. This is purely based on the steps spans themselves (Not the Page: ___ spans that are created on frame change). i.e. pdp_page_load would be counted as a PDP page view for us as it takes us to the page.

We can see we have:

Now we check in Google Analytics to verify the page views, and we get very different numbers there for our page views (2919 total)

The data from GA is much more like the split we are expecting to see, and have subsequently configured.

Here is also the graph that is using the Page: __ spans created by artillery on frame change, and the numbers are even lower than what we expected.

Screenshot 2024-09-19 at 14 31 55

I've ran multiple iterations of the test utilising either DD directly or OTEL and the results are always very similar - way off what is expressed in GA. So I think we are missing quite a lot of spans

We do a series of steps and transactions which will cause a mixture of hard and soft navigations across our site, could this potentially cause any issues with recording traces for steps and scenarios?