Closed arthuralee closed 1 year ago
hey @arthuralee, thanks for the report! This is indeed a bug. You're right in considering that test.command
should not be part of the test.full_name
. It isn't, but test.bundle
is, but it turns out test.bundle
is not being handled correctly.
I'll push a fix as soon as possible.
While implementing test visibility for jest tests, I found that a flaky test was not properly recognized by Datadog. The underlying issue was that the
test.full_name
tag for the two runs did not match, even though they were the same test.This is because the jest/CI trace plugins send the
command
as the test module/bundle/command, which is then used as a prefix for the test full_name. In our setup, we may dynamically determine the command that we use to invoke jest, possibly resulting in a different command. For example, one run may runjest ./specA.spec.ts ./specB.spec.ts
, while a second retry run may only runjest ./specA.spec.ts
. As a result, even though the same tests in specA are being run, Datadog recognizes them as different tests due to the differing commands.Expected behaviour The same test in the same suite in the same spec file should be recognized as the same test even if they are run in combination with different specs. Perhaps the "command" should not be part of the full_name at all, but i don't have the full context on why it was added
Actual behaviour The same test in the same suite in the same spec file is not recognized as the same test if the args passed to jest are different
Steps to reproduce
specA.spec.ts
andspecB.spec.ts
NODE_OPTIONS='-r dd-trace/ci/init' jest specA.spec.ts
NODE_OPTIONS='-r dd-trace/ci/init' jest specA.spec.ts specB.spec.ts
test.full_name
sEnvironment