search

hcoles / pitest

State of the art mutation testing system for the JVM
Apache License 2.0
1.7k stars 357 forks source link

Some test methods cannot be found in mutation test matrix #535

Open tatianguiqu opened 6 years ago

tatianguiqu commented 6 years ago

I try to get full mutation matrices for several projects, but some test methods cannot be found in quite a part of the projects. The configuration for the plugin looks like this: <plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <version>1.4.3</version> <configuration> <fullMutationMatrix>true</fullMutationMatrix> <outputFormats>XML</outputFormats> </configuration> </plugin> One example of these XML reports, each part consists of two identical class names in the last line of the tag "killingTests": image There are many such situations that prevent us from getting test methods to make full mutation matrix. Those projects that have such anomalies such as Java-WebSocket, jsoup, JSqlParser, etc. can be downloaded from GitHub.

hcoles commented 6 years ago

@nrainer can you take a look at this?

nrainer commented 5 years ago

@tatianguiqu Can you provide more details? Which project in which revision is this? What is the name of the test case that you expected to appear in the matrix but is not there? Are you aware of any special test runners that are used?

tatianguiqu commented 5 years ago

@nrainer One example： project Java-WebSocket version 1.3.9 One test class: org.java_websocket.protocols.ProtoclHandshakeRejectionTest There are many test cases in this class: image I want to get these test cases names in the matrix, but there is only class name in it: image A normal test case in the XML consists of method name and class name, where the class name is inside a bracket : image There is no bracket and no method name in these abnormal ones. And it seems that any test case in the test class which kill the mutation will generate one block just like the second picture, so I can find a very long line consists of all the same blocks.

nrainer commented 5 years ago

I was debugging https://github.com/TooTallNate/Java-WebSocket in revision 587a0adb27d7b2f806554b818fe8c8c3971514ed.

The test string for the killed matrix for the first or second executed mutation [1] involving the test class ProtoclHandshareRejectionTest looks as follows: org.java_websocket.drafts.Draft_6455Test.acceptHandshakeAsClient(org.java_websocket.drafts.Draft_6455Test)|org.java_websocket.issues.Issue609Test.testIssue(org.java_websocket.issues.Issue609Test)|org.java_websocket.issues.Issue677Test.testIssue(org.java_websocket.issues.Issue677Test)| <...> |org.java_websocket.protocols.ProtoclHandshakeRejectionTest.org.java_websocket.protocols.ProtoclHandshakeRejectionTest|org.java_websocket.protocols.ProtoclHandshakeRejectionTest.org.java_websocket.protocols.ProtoclHandshakeRejectionTest|org.java_websocket.protocols.ProtoclHandshakeRejectionTest.org.java_websocket.protocols.ProtoclHandshakeRejectionTest.

Hence, it indeed includes strange test identifiers, such as, the mentioned org.java_websocket.protocols.ProtoclHandshakeRejectionTest.org.java_websocket.protocols.ProtoclHandshakeRejectionTest.

@hcoles Since the string for the mutation matrix contains both valid and strange test identifiers, I don't think that this issue is related to the mutation matrix. The strange identifiers are not created by the matrix, but arrive already like this in CheckTestHasFailedResultListener. I assume that they might be caused by the timeout specified at the test cases (@Test(timeout = 5000), see screenshot above), which causes that the tests are executed in another way. However, I was not able to reproduce this problem on a project with a single test class with tests with timeouts using junit-4.11 (which is the same version as used in this project). The problem might be a combination of timeouts + junit version + surefire version.

[1] MutationIdentifier [location=Location [clazz=org.java_websocket.drafts.Draft_6455, method=acceptHandshakeAsClient, methodDesc=(Lorg/java_websocket/handshake/ClientHandshake;Lorg/java_websocket/handshake/ServerHandshake;)Lorg/java_websocket/drafts/Draft$HandshakeState;], indexes=[160], mutator=org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator]

nrainer commented 5 years ago

In order words: I think that org.pitest.testapi.Description.getQualifiedName() is returning an unexpected string for these tests.