Introduce mutation testing to Eclipse Collections build

[donraab]

We should have mutation tests running in the Eclipse Collections build so we can discover and address more edge cases in our tests.

Pitest

[prathasirisha]

Please assign this ticket to me @donraab

[donraab]

Thanks for volunteering @prathasirisha ! Once you commented here, you showed up in the assignee list so I assigned.

[prathasirisha]

Requesting guidance on how to proceed with this issue @nikhilnanivadekar @donraab @motlin

Context- We have a total of 150k tests in unit-tests module in EC.

1. Upon running mutation tests as it is (for all ~150k tests), we run into OutOfMemory errors.
2. Upon running mutation tests using the configuration from my initial commit which doesn't reduce any tests, we ran into same issue as #1
3. With Don's help we reduced the number of tests to 10k (uncommitted changes atm) and we were able to get a successful mutation test run which took about an hour to run.

Questions-

1. Do we still want to enable mutation testing for the ~10k tests ?

   • Pros: No OutOfMemory issues Time taken is less compared to running pitest for ~150k tests
   • Cons: Reduced scope

2. Do we create separate builds for each of the sub folders like bag, bimap, set, etc. to reduce the number of tests running in a single build? Or any other grouping logic for that matter.

   • Pros: Successful 100% mutation test coverage
   • Cons: Time consuming Not sure if we should even be running this for after every commit.

3. Do we identify top contenders that need this testing in place and only run the mutation tests for them?

   • Pros: Successful mutation test coverage for "important" classes. Overall time taken will be less compared without running into OutOfMemory issues
   • Cons: Reduced Scope

4. Or if there are any other ideas from the group.

I can share the output of the pitest if anyone is interested from #1 run.

[nikhilnanivadekar]

The out of memory error happens for how much JVM memory? Is there a way to bump that up? We can run this in TeamCity which gives us higher configuration build runners if that’ll help

[donraab]

I ran with 16gigs of RAM and it did not help.

[motlin]

With mutation testing, you get a lot of value from running the tests once. When you got them running, what did you find? Any failures?

On Sun, Apr 12, 2020 at 9:05 PM Nikhil Nanivadekar notifications@github.com wrote:

The out of memory error happens for how much JVM memory? Is there a way to bump that up? We can run this in TeamCity which gives us higher configuration build runners if that’ll help

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/eclipse/eclipse-collections/issues/807#issuecomment-612704330, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB3UIULD3WWMQKEHDHXBM3RMJQL5ANCNFSM4K7IBZGA .

[donraab]

I think we should ask @prathasirisha to submit a PR and set up a GitHub Actions build for the configuration we got working so we can review the results. Make sense?

[nikhilnanivadekar]

Yup that’ll be better for review and observation.

[prathasirisha]

Sounds good. I will do as suggested by @donraab and we can review the results then.

[donraab]

Excellent, thank you @prathasirisha!

[prathasirisha]

• Committed changes that worked for getting a successful mutation test run.
• Set up mutation testing workflow in my personal fork. Build here You should see "Artifacts" on top right of this build to download the pitest generated reports for coverage.

Please review them - @nikhilnanivadekar @motlin @donraab