[Question] Merging code coverage reports - Out of memory stuck

aldoreyes commented 2 days ago

We are trying to merge the code coverage reports generated in an azure pipeline by multiple jobs (playwright shards). Since the azure pipeline hosted agents are quite limited on the memory (~6GB) we get constantly to a point where the merge process of the monocart library gets stuck when we reach 100% of memory usage in the agent. I know we could try using a self hosted solution with an increased memory limit, but we were wondering if there is something that could do without incrementing our costs.

Our current pipeline setup is 5 jobs processing the sharded tests. Which usually generate code coverage reports of around ~60-100MB. After these we execute a job that downloads the artifacts and execute the merge process using the next cli command.

monocart merge monocart-report/**/index.json -c mr.config.js

Just as extra info, our current setup works perfectly on our local machines where we get the merged reports correctly.

Thanks for an awesome library!

cenfun commented 1 day ago

Possible solution is merge coverage files for raw report, see https://github.com/cenfun/monocart-reporter/issues/151

// coverage options
{
  reports: [
    ["raw", {
      merge: true,
      zip: true
    }]
  ]
}

how many coverage files generated in each of shards?

try option logging: "debug" to show the details of memory

[MCR] added from dir: ./coverage/v8-coverage (memory: 92.1MB) (5.5s)
[MCR] ┌ [generate] prepared coverage data (memory: 93.2MB) (5.7s)
[MCR] ┌ [generate] merged v8 coverage data (memory: 99.6MB) (65ms)
[MCR]   ┌ [convert] parsed ast: build/bin.js (186 B) (memory: 99.9MB) (4ms)
[MCR]   ┌ [convert] unpacked sourcemap: build/bin.js (1 files) (memory: 100.1MB) (1ms)
[MCR]   ┌ [convert] parsed ast: build/tstyche.js (158.1 KB) (memory: 96.2MB) (48ms)
[MCR]   ┌ [convert] unpacked sourcemap: build/tstyche.js (93 files) (memory: 128.6MB) (88ms)
[MCR]   ┌ [convert] converted data list (94 files) (memory: 136.2MB) (18ms)
[MCR] ┌ [generate] converted coverage data (memory: 136.2MB) (179ms)

Or try node option --max-old-space-size
```
NODE_OPTIONS=--max-old-space-size=4096
```

aldoreyes commented 1 day ago

@cenfun thanks for your answer, I tried using those options but we are getting the same problem where the usage of memory goes to full, here are the last pieces of the logs.

portal:test:e2e:merge: [MCR]   ┌ [convert] unpacked sourcemap: localhost-3000/_next/static/chunks/main-app.js/v=1728165951069 (147 files) (memory: 9167.6MB) (796ms)
portal:test:e2e:merge: [MCR]   ┌ [convert] parsed ast: localhost-3000/_next/static/chunks/webpack.js/v=1728165957006 (54.4 KB) (memory: 9160.1MB) (40ms)
portal:test:e2e:merge: [MCR] source filter (localhost-3000/_next/static/chunks/webpack.js/v=1728165957006): before 28 => after 0
portal:test:e2e:merge: [MCR] added source files: 0
portal:test:e2e:merge: [MCR]   ┌ [convert] unpacked sourcemap: localhost-3000/_next/static/chunks/webpack.js/v=1728165957006 (0 files) (memory: 9163.1MB) (4ms)
portal:test:e2e:merge: [MCR]   ┌ [convert] parsed ast: localhost-3000/_next/static/chunks/main-app.js/v=1728165957006 (2.8 MB) (memory: 9237.3MB) (932ms)
portal:test:e2e:merge: [MCR] source filter (localhost-3000/_next/static/chunks/main-app.js/v=1728165957006): before 180 => after 147
portal:test:e2e:merge: [MCR] added source files: 147
portal:test:e2e:merge: [MCR]   ┌ [convert] unpacked sourcemap: localhost-3000/_next/static/chunks/main-app.js/v=1728165957006 (147 files) (memory: 9287.6MB) (1.2s)
portal:test:e2e:merge: [MCR]   ┌ [convert] parsed ast: localhost-3000/_next/static/chunks/webpack.js/v=1728165911146 (54.4 KB) (memory: 9290.3MB) (23ms)
portal:test:e2e:merge: [MCR] source filter (localhost-3000/_next/static/chunks/webpack.js/v=1728165911146): before 28 => after 0
portal:test:e2e:merge: [MCR] added source files: 0
portal:test:e2e:merge: [MCR]   ┌ [convert] unpacked sourcemap: localhost-3000/_next/static/chunks/webpack.js/v=1728165911146 (0 files) (memory: 9293.2MB) (21ms)

These are the last logs after this the logs don't print anything and after 10 minutes in this state the agent gets canceled due to not responding. Also this is a nextjs app in case that helps.

Each shard generates around ~30 files with the prefix coverage-.json and another ~30 files with the prefix source-.json

cenfun commented 21 hours ago

It should be crashed with a error message if OOM

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Freeze for 10 minutes seems that something is causing 100% CPU usage? Im not sure.

Based on the last pieces of the logs, I can't find where the problem is. Could you please provide a minimal reproducible repo? So I can reproduce and debug the issue locally? (Or upload the monocart-report/**/index.json, all coverage json files and mr.config.js to somewhere and let me download)

cenfun commented 21 hours ago

here is a big project TypeScript, the max memory reached 1154.7MB but without OOM.

[MCR] added coverage data: v8 (memory: 249.6MB) (345ms)
[MCR] added coverage data: v8 (memory: 195.6MB) (78ms)
[MCR] added coverage data: v8 (memory: 316.3MB) (72ms)
[MCR] added coverage data: v8 (memory: 422.4MB) (69ms)
[MCR] added coverage data: v8 (memory: 5[47](https://github.com/cenfun/monocart-coverage-reports/actions/runs/11198037519/job/31128756271#step:8:48).9MB) (78ms)
[MCR] added from dir: /home/runner/work/monocart-coverage-reports/monocart-coverage-reports/coverage/tmp (memory: 547.9MB) (1.3s)
[MCR] ┌ [generate] prepared coverage data (memory: 547.9MB) (1ms)
[MCR] ┌ [generate] merged v8 coverage data (memory: 571.9MB) (209ms)
[MCR]   ┌ [convert] parsed ast: built/local/run.js (12.4 MB) (memory: 426.4MB) (6.4s)
[MCR]   ┌ [convert] unpacked sourcemap: built/local/run.js (590 files) (memory: 1154.7MB) (6.5s)
[MCR]   ┌ [convert] converted data list (590 files) (memory: 1004.2MB) (1.9s)
[MCR] ┌ [generate] converted coverage data (memory: 1004.2MB) (14.9s)
[MCR] ┌ [generate] saved report: lcovonly (memory: 1018.9MB) (732ms)
[MCR] ┌ [generate] saved report: cobertura (memory: 1026MB) (916ms)
[MCR] ┌ [generate] saved report: v8 (memory: 1174MB) (1.1s)
[MCR] ┌ [generate] saved report: v8-json (memory: 319.8MB) (369ms)
[MCR] ┌ [generate] saved report: codecov (memory: 331MB) (29ms)
[MCR] generated coverage reports: coverage/index.html (memory: 331MB) (18.2s)

see https://github.com/cenfun/monocart-coverage-reports/actions/runs/11198037519/job/31128756271

and another project tstyche which have 300+ coverage files generated without OOM see https://github.com/cenfun/monocart-coverage-reports/actions/runs/11198037151/job/31128756126

aldoreyes commented 1 hour ago

Thanks for the quick replies @cenfun, I don't think I can provide the source code since is a private project. I'll continue debugging this, is strange that our files need so much memory compared to yours

portal:test:e2e:merge: [MCR] ┌ [convert] parsed ast: localhost-3000/_next/static/chunks/webpack.js/v=1728165911146 (54.4 KB) (memory: 9290.3MB) (23ms)

Probably something with nextjs and how we merge the backend/frontend reports on each shard that may be duplicating things or something like that. Again thanks for checking on this and I'll get back if I found the cause on our side.

cenfun / monocart-reporter

[Question] Merging code coverage reports - Out of memory stuck #153