search

STAMP-project / pitest-descartes

Descartes supports developers to improve their test suites by reporting weak spots in covered code
https://hal.archives-ouvertes.fr/hal-01870976/document
GNU Lesser General Public License v3.0
126 stars 21 forks source link

Junit5 @AfterAll does not seems to be taken into account #106

Closed tmortagne closed 4 years ago

tmortagne commented 5 years ago

Characteristics

Description

The following test case https://github.com/xwiki/xwiki-platform/blob/b63f7cc3c722325d4a3e93ab97015e7122c95603/xwiki-platform-core/xwiki-platform-mailsender/src/test/java/com/xpn/xwiki/plugin/mailsender/MailSenderApiTest.java start a smtp test server in @BeforeAll and stop it in @AfterAll and execute a few tests in between.

When I execute the build with pitest enabled I can see a lot of failing greenmail startup because the port is already used (i.e. it's already running) as if @BeforeAll was called but not @AfterAll or if @BeforeAll was executed for each test instead of only once.

Steps to reproduce

Build module https://github.com/xwiki/xwiki-platform/blob/b63f7cc3c722325d4a3e93ab97015e7122c95603/xwiki-platform-core/xwiki-platform-mailsender with 

mvn clean install -Pquality -Dxwiki.pitest.skip=false -Djacoco.skip=true
tmortagne commented 5 years ago

Tried with pitest-junit5-plugin 0.9 but not better.

oscarlvp commented 5 years ago

When I try to build xwiki-plattform I get the following error:

Exception in thread "DefaultMetadataResolver-44-0" Exception in thread "DefaultMetadataResolver-44-1" java.lang.NoSuchMethodError: org.eclipse.aether.transfer.TransferResource.<init>(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/io/File;Lorg/eclipse/aether/RequestTrace;)V

Any idea of what am I missing?

tmortagne commented 5 years ago

That's weird, does not seems to be a XWiki thing. What version of Maven are you using ? Could you past the full error ?

tmortagne commented 5 years ago

Also don't try to build the whole xwiki-platform (takes ages), just make sure you have XWiki SNAPSHOTS repository in your settings.xml (see https://dev.xwiki.org/xwiki/bin/view/Community/Building/#HInstallingMaven) and just build xwiki-platform-mailsender module.

oscarlvp commented 5 years ago

I'm using Maven 3.5.0. I will try to build only xwiki-platform-mailsender.

tmortagne commented 5 years ago

We usually build with 3.6 since a little while, maybe there is issues with 3.5.

oscarlvp commented 5 years ago

Ah, OK, then I'll update my Maven version and try again.

vmassol commented 5 years ago

@oscarlvp The issue is still there. Have you been able to reproduce?

Just ran it and got:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  37.764 s
[INFO] Finished at: 2019-09-20T09:29:44+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.pitest:pitest-maven:1.4.10:mutationCoverage (pitest-check) on project xwiki-platform-mailsender: Mutation score of 9 is below threshold of 36 -> [Help 1]

This is lowering our mutation score a lot and failing our build.

Side note/idea: it could be great if you had some regression test suite for Descartes so that you could verify that each release doesn't decrease mutation score because otherwise it's a pain for projects like XWiki who fail the build on reduced mutation scores ;) It doesn't send good signals to devs when there are false positives as it decreases their trust in the tool (and we don't want that! :)).

EDIT: it could actually well be an issue with PIT and not Descartes.

EDIT 2: I just understood that it's not a regression and that the issue has probably been always there. Still the "side note/idea" above is a good one IMO:)

vmassol commented 4 years ago

Seems it's working fine now ;) (with v0.12 of the junit5 pitest plugin).