Slow performances when running tests with `--experimental-test-coverage`

rozzilla commented 23 hours ago

Version

v22.9.0

Platform

Darwin N4V4PGFGPT 23.6.0 Darwin Kernel Version 23.6.0: Mon Jul 29 21:16:46 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T8112 arm64

Subsystem

No response

What steps will reproduce the bug?

Run tests with --experimental-test-coverage option. Comparing it to other solutions (f.e. running Node tests and getting the coverage with nyc) this is way slower. For instance, on a repo with ~800 tests, collecting coverage with nyc takes 77s, while running with --experimental-test-coverage takes 148s.

How often does it reproduce? Is there a required condition?

I can reproduce it every time I run the Node tests with the native test coverage option.

What is the expected behavior? Why is that the expected behavior?

The performances should be at least equal (ideally better) than with 3rd party dependencies (like nyc).

What do you see instead?

An average time spent per test of 185ms vs 96ms.

Additional information

I tried running with/without additional options (f.e. --test-coverage-exclude, --test-coverage-include, --test-coverage-branches, --test-coverage-functions, --test-coverage-lines) but it didn't help. The performance issue seems only to be related to the algorithm that's triggered by the --experimental-test-coverage option.

cjihrig commented 23 hours ago

Did you happen to look into where the time is being spent? For example, do you know if most of the time is being spent collecting the coverage, or in parsing it to generate the report? If the time is being spent collecting the coverage, there likely isn't much we can do. Another thing you can try is collecting coverage data without using the test runner's coverage feature.

rozzilla commented 23 hours ago

Did you happen to look into where the time is being spent? For example, do you know if most of the time is being spent collecting the coverage, or parsing it to generate the report?

Looking closely at the test execution, it seems that both things are happening:

execution time per single test is higher (~2x times) when running with coverage option
parsing and generating the report takes a lot of time, in fact, the test runner seems stuck for a bunch of seconds before generating the report, and then finally it outputs the result

To summarize, my assumption is that the performance issue seems to be a combination of both factors you mentioned @cjihrig. 👍🏼

cjihrig commented 22 hours ago

A few questions:

Is this codebase public by any chance?
Are you using source maps?
What does the execution time look like if you run Node with NODE_V8_COVERAGE, but not the --experimental-test-coverage flag?

rozzilla commented 2 hours ago

Is this codebase public by any chance?

Nope, but to give some more context about it, it's a fastify app exposing endpoints that do internal HTTP calls, query a PG Database, get cached data from Redis, etc. The repository has both unit tests, and integration tests that check everything (apart from mocking HTTP calls, and some other edge cases).

Are you using source maps?

Nope

What does the execution time look like if you run Node with NODE_V8_COVERAGE, but not the --experimental-test-coverage flag?

It's slightly slower (~5s)

nodejs / node