Merging coverage profiles leads lots of duplicated lines and uses lots of memory

[myzhan]

I'm using ginkgo to run the testcases in a project that has 26w lines of code, with the -p flag enabled.

Issues

1. When running testcases in parallel, more cpu cores lead to more duplicated lines. If I add the -p flag and run on a 64 cores machine, the size of cover.out file is 7.9GB. And the size of cover.out file is 130MB when -p flag removed.
2. All the coverage files are read into combined buffer, which use lots of memory. And ginkgo will be oom-killed.

[onsi]

hey there, thanks for reporting this.

would you be up for trying to fix this? the code that does this currently lives here - it loads everything into memory and simply concatenates them into one massive file.

gocovmerge shows how to do this correctly. we can either take a dependency on gocovmerge and reimplement MergeAndCleanupCoverProfiles using it, or we can reimplement its merge logic in Ginkgo directly. Either is fine with me - though using it seems more straightforward. You shouldn't need any new tests as there is ample integration test coverage for mreging cover files.

let me know if you'd be up for workin on a PR.

[myzhan]

OK, I'll find some time to submit a PR.