Merging coverage for same file is non deterministic

sterling commented 3 years ago

Reproducible bug demo: https://github.com/sterling/nyc-bug

Two coverage json reports are provided in the demo under coverage/. These were generated by jest and karma, and cover a single source file in different ways. Merging the two reports results in a different final report when run multiple times. The easiest way to reproduce is by running nyc report -t coverage --reporter=text-summary many times and comparing the output.

Expected Behavior

nyc merges coverage files consistently

Observed Behavior

nyc merges coverage files differently on subsequent runs. I have reproduced on multiple developers' machines, but in some cases it required up to 20 runs before reproducing (possibly due to slower machines).

Troubleshooting steps

I believe this is caused by the concurrent nature of merging reports. By changing the concurrency of merging files to 1, it seems to resolve my issue: https://github.com/istanbuljs/nyc/blob/master/index.js#L430

Environment Information

  System:
    OS: macOS 11.1
    CPU: (16) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 587.69 MB / 32.00 GB
  Binaries:
    Node: 12.20.1 - /usr/local/bin/node
    Yarn: 1.22.10 - /usr/local/bin/yarn
    npm: 6.14.10 - /usr/local/bin/npm
  npmPackages:
    nyc: ^15.1.0 => 15.1.0

sieukrem commented 3 years ago

@sterling can you check the linked PR if it solves this issue?

pkarman commented 2 years ago

@sieukrem https://github.com/istanbuljs/istanbuljs/pull/601 appears to have been closed without merging but there is reference to a "recent release". We are experiencing this same issue with merging reports and would love to try the newest release but I'm not sure, exactly, which library is supposed to contain the fix. Can you point me to which npm package I need to update?

sieukrem commented 2 years ago

@pkarman I have updated listed below dependencies

    "istanbul-lib-coverage": "3.2.0",
    "istanbul-lib-instrument": "5.1.0",
    "istanbul-lib-source-maps": "4.0.1",
    "istanbul-reports": "3.0.5",

istanbuljs / nyc