6GB ram is not enough to evaluate 10 million unit test runs?

[Speiger]

o/ Part 2 to this issue. I hope you remember https://github.com/Speiger/Primitive-Collections/runs/7214885381?check_suite_focus=true

After speaking to @dsame the guy who is maintaing the SetupJava Action (Hope you don't mind to be included), we well more he figured out how to remove all the memory usage left behind by gradle, and now 6GB are left in the virtual machine. I appended 4GB of Swap Memory into the script and this almost let this action complete, but sadly it wasn't enough.

So, at this point I think this action is just from a concept/implementation not capable of dealing with that many unit tests. Which is really sad because I wanted to use the full capability of this feature to provide as much data as possible with it.

I mainly open this issue because I have given up on this. But if you want to improve your tool to handle that much data this issue is basically my offer to test it when you need something to test against.

But also to kinda give a report back that this has massive limitations sadly.

Anyhow thank you for improving your logging so we can at least see a tiny bit more info of what is going wrong and good luck with making improvements if you decide to do so.

[EnricoMi]

I have reproduced your test result files. They contain 1.8 GB worth of test names and 0.7 GB of class names. Those strings would have to be stored in RAM, probably multiple times (one complete list, once indexed, ...).

Those names would have to be sent to GitHub as annotations for every workflow run in order to use the full capability of this feature. I think this is unrealistic (from GitHub API point of view. So check_run_annotations has to be none to not send that vast of information to GitHub API.

I could make the action drop all test cases that succeed, so only those are memorized that fail or are skipped. This would disable the feature of displaying individual run information but would still allow to get failure annotations, which is probably the most useful feature of this action.

[Speiger]

How about a packages only mode? Where you can define a "package" (if not defined it goes back to the current behavior) and it sorts them by package? since i have like 7 packages where everything comes down to that would make sense. And if that makes sense maybe a package depth how deep from the base package it should filter.

I mean the counting of tests runs.

[EnricoMi]

You get the counting of test results with ignore_runs: true already.

The memory-expensive bit is identifying multiple runs of the same test and detecting added / removed tests.

[ben-manes]

A sketch, e.g. cuckoo filter or countmin, might solve that at a low enough error rate (estimate). I’m quite happy with the current behavior, though.

[Speiger]

You get the counting of test results with ignore_runs: true already.

The memory-expensive bit is identifying multiple runs of the same test and detecting added / removed tests.

Oh yeah it does that. But it is kinda interesting to see what package has more tests from others. Like seeing it is not required. But visualizing even testing data like that is beautiful.

Thats why it is so sad that the project is to large :/

But oh well you can't always have your cake and eat it too xD

[EnricoMi]

You could run the action once for each of the sub-directories in build/test-results/. Then the content is down to 400 MB for each action run. You would still need set check_run_annotations: none, but would get proper failure annotations.

With comment_mode: failures, you could make the actions less chatty and only report failures.