Support large test suites

[ben-manes]

My build runs ~9M tests per matrix item, which results in very large junit test results. When experimenting, this action failed due heap exhaustion.

I am currently using publish-unit-test-result-action which had this issue. Prior to their fix I used some bash to truncate with,

find . -type f -name '*.xml' -exec sh -c 'grep testsuite {} > {}.out && mv {}.out {}' \;

[ben-manes]

That trick didn’t work, unfortunately. The parser did not take the summary line and instead found no tests.

[ethomson]

Yikes. Thanks for opening the issue @ben-manes. The parser that I used is pretty naive, it was a quick & dirty attempt. I can switch to a streaming parser and that should be able to parse very large XML documents.

There's still an opportunity to exhaust the heap if you had (say) 9M test failures, each with a very long stack trace or something. We should be thoughtful of that as well and put a cap on the number of failures that we collect for showing the output table and truncate beyond that.

As for why your truncation didn't work, I was examining each testcase since my understanding is that failures and skipped are optional in the XML. However, rethinking this, I suspect that what they mean is that the failure property is optional and if omitted means that there are no failures. I thought that I had found a reporter that would omit the failures property, but now I'm suspicious of my memory. I'll go through this again. If every reporter in common usage always shows a failures property on a testsuite when there are actually failures, then we don't need to bother with testcases unless you want more detailed output. So a mode that doesn't show the test cases table would need not parse testcases at all. This would be a quick & dirty solution that would make your truncation work.

[crazy-max]

I'm experimenting with a matrix myself and it seems to work fine: https://github.com/crazy-max/buildkit/blob/dc1335ba667df6977e9b02440566c77b297632c7/.github/workflows/build.yml#L224-L275

Result: https://github.com/moby/buildkit/actions/runs/2498837572#summary-6895289837

I'm also using another tool called teststat to have other useful info. Might be cool to have it here too.

[npepinpe]

We're having the same issue with some builds triggering resource exhaustion due to very large matrix. See for example this run: https://github.com/camunda/zeebe/actions/runs/7895500367/job/21547941014

Couple things that could factor: test result files for these tend to be huge (like ~350MB), but generally the number of tests is low-ish (like < 1k).