search

szpak / gradle-pitest-plugin

Gradle plugin for PIT Mutation Testing
http://gradle-pitest-plugin.solidsoft.info/
212 stars 57 forks source link

report aggregation process implemented for multi-project #243

Closed MikeSafonov closed 3 years ago

MikeSafonov commented 3 years ago

Greetings I have implemented a report aggregation process using pitest-aggregator

MikeSafonov commented 3 years ago

Hello. Any updates?

szpak commented 3 years ago

Hello. Any updates?

Unfortunately, I was busy with some other activities. Nevertheless, I will try to handle that "soon".

szpak commented 3 years ago

I wanted to work on it, however, it still fails for me locally with commit 140af32ce2c5ff9eebb990d1153c3120eb32ece8 (your master):

info.solidsoft.gradle.pitest.functional.AcceptanceTestsInSeparateSubprojectFunctionalSpec > should aggregate report from subproject FAILED
    java.lang.IllegalArgumentException: Mutable Project State warnings were found (Set the ignoreMutableProjectStateWarnings system property during the test to ignore):
     - The configuration :for-report:testRuntimeClasspath was resolved without accessing the project in a safe manner.  This may happen when a configuration is resolved from a different project. This behaviour has been deprecated and is scheduled to be removed in Gradle 7.0. See https://docs.gradle.org/6.6/userguide/viewing_debugging_dependencies.html#sub:resolving-unsafe-configuration-resolution-errors for more details.
     - The configuration :shared:testRuntimeClasspath was resolved without accessing the project in a safe manner.  This may happen when a configuration is resolved from a different project. This behaviour has been deprecated and is scheduled to be removed in Gradle 7.0. See https://docs.gradle.org/6.6/userguide/viewing_debugging_dependencies.html#sub:resolving-unsafe-configuration-resolution-errors for more details.
        at nebula.test.IntegrationBase$Trait$Helper.checkForMutableProjectState(IntegrationBase.groovy:114)
        at nebula.test.Integration$Trait$Helper.checkForDeprecations(Integration.groovy:171)
        at nebula.test.Integration$Trait$Helper.runTasks(Integration.groovy:167)
        at nebula.test.Integration$Trait$Helper.runTasksSuccessfully(Integration.groovy:148)
        at info.solidsoft.gradle.pitest.functional.AcceptanceTestsInSeparateSubprojectFunctionalSpec.should aggregate report from subproject(AcceptanceTestsInSeparateSubprojectFunctionalSpec.groovy:31)

21 tests completed, 1 failed, 3 skipped

When I ignored the deprecation warnings it started to fail with:

                Caused by:
                java.lang.IllegalStateException: Failed to build: no lineCoverageFiles have been set
                    at org.pitest.aggregate.ReportAggregator$Builder.validateState(ReportAggregator.java:226)
                    at org.pitest.aggregate.ReportAggregator$Builder.build(ReportAggregator.java:214)
                    at info.solidsoft.gradle.pitest.report.PitestReportAggregator.aggregate(PitestReportAggregator.groovy:38)

For some reasons the CI builds pass. I made a dummy commit in your branch to make sure Travis builds from the same commit, but it stuck (most likely due to that).

Does ./gradlew check funcTest work for you locally? What OS and Java version do you use (I checked with Java 15, 11 and 8 on Linux)?

Update. You seem to migrate already to travis.com and that test passed :-/ (the fail on upload is related to my CD mechanism and you may ignore it - I will apply a workaround before the next version is released)

szpak commented 3 years ago

I'm not sure why it was happening only on my machine, but both "no lineCoverageFiles" and deprecation warnings were caused by the fact that in functional tests pitestReportAggregate was executed before the regular pitest tasks. In that situation, there were not files with line coverage which was generating the errors. I added mustRunAfter (mysteriously shouldRunAfter was not enough :thinking: ) which fixed the test.

That brought me to the conslusion that AggregateReportTask has no inputs and outputs, it will be always not up-to-date (even there are not changes in previously generated by pitest tasks reports). Woudn't be possible to try to expose them as fields in AggregateReportTask?

Unfortunately, solving that took a lot of time, so I will experiment with avoiding hardcoded versions of runtime dependencies another time :-/.

MikeSafonov commented 3 years ago

Greetings. I`ve revorked AggregateReportTask to use Output and Intput

szpak commented 3 years ago

My initial approach with compileOnly dependency (and runtime hacks) failed. I asked for other suggestions on their forum.

szpak commented 3 years ago

With a help from @MarcPhilipp, I was able to retain the PIT version configuration at runtime capability using the Worker API.

@MikeSafonov I had to rework your PR a little bit. Please take a look if it looks ok for you (the last commit). I will write some more tests tomorrow and I plan to merge it soon. Feel free to make new commits if needed, but please do not rebase this branch.

MikeSafonov commented 3 years ago

With a help from @marcphilipp, I was able to retain the PIT version configuration at runtime capability using the Worker API.

@MikeSafonov I had to rework your PR a little bit. Please take a look if it looks ok for you (the last commit). I will write some more tests tomorrow and I plan to merge it soon. Feel free to make new commits if needed, but please do not rebase this branch.

Congratulations! I have already looked at your changes and they looks good to me. Looking forward to the new version of the plugin!

szpak commented 3 years ago

I've merged it manually into devel and having some other (unrelated) changes implemented I will release 1.6.0 "soon". Thank you for your contribution!