search

DataDog / dd-trace-js

JavaScript APM Tracer
https://docs.datadoghq.com/tracing/
Other
647 stars 306 forks source link

Jest/CI plugin for test visibility - test uniqueness problem #2874

Closed arthuralee closed 1 year ago

arthuralee commented 1 year ago

While implementing test visibility for jest tests, I found that a flaky test was not properly recognized by Datadog. The underlying issue was that the test.full_name tag for the two runs did not match, even though they were the same test.

This is because the jest/CI trace plugins send the command as the test module/bundle/command, which is then used as a prefix for the test full_name. In our setup, we may dynamically determine the command that we use to invoke jest, possibly resulting in a different command. For example, one run may run jest ./specA.spec.ts ./specB.spec.ts, while a second retry run may only run jest ./specA.spec.ts. As a result, even though the same tests in specA are being run, Datadog recognizes them as different tests due to the differing commands.

Expected behaviour The same test in the same suite in the same spec file should be recognized as the same test even if they are run in combination with different specs. Perhaps the "command" should not be part of the full_name at all, but i don't have the full context on why it was added

Actual behaviour The same test in the same suite in the same spec file is not recognized as the same test if the args passed to jest are different

Steps to reproduce

  1. Create a repo with two jest test files, specA.spec.ts and specB.spec.ts
  2. Run NODE_OPTIONS='-r dd-trace/ci/init' jest specA.spec.ts
  3. Run NODE_OPTIONS='-r dd-trace/ci/init' jest specA.spec.ts specB.spec.ts
  4. Observe in the Datadog traces that the tests in specA that have run twice have different test.full_names

Environment

juan-fernandez commented 1 year ago

hey @arthuralee, thanks for the report! This is indeed a bug. You're right in considering that test.command should not be part of the test.full_name. It isn't, but test.bundle is, but it turns out test.bundle is not being handled correctly.

I'll push a fix as soon as possible.