search

hcoles / pitest

State of the art mutation testing system for the JVM
Apache License 2.0
1.71k stars 357 forks source link

Failure in initializing parameterized tests would lead to wrong full mutation matrix result #1046

Open qinfendeheichi opened 2 years ago

qinfendeheichi commented 2 years ago

If we write parameterized tests in Junit5, and if there exists a static method that provides values for parameterized tests, and if any value to be produced somehow crashes in the process

Then, all parameterized test cases would be reported as "failed" because the method that provides values fails, though maybe only 1 test case should've failed.

Hence, it does not hurt the mutation score or test strength in most cases, but it hurts the real mutation matrix score. It seems that executing the whole method is inevitable for such parameterized tests.

ThomHurks commented 2 years ago

We have one project where upon upgrading to pitest 1.9.0 the mutation score dropped from 100 to around 50. The project uses a lot of parameterized Junit5 tests, so I think in general 1.9.0 may have introduced a regression when it comes to parameterized tests?

hcoles commented 2 years ago

@ThomHurks If you can post a project that reproduces the issue, I can take a look.