search

hcoles / pitest

State of the art mutation testing system for the JVM
Apache License 2.0
1.67k stars 358 forks source link

include mutated code in report #146

Closed UrsMetz closed 5 months ago

UrsMetz commented 9 years ago

Currently the mutations are only described in textual form which makes it sometimes hard to find out which kind of mutation of the code is not covered by the tests. I think including the mutated code (like mutant does in the report in the terminal) for survived mutations in the HTML reports created by PIT could help with that. This would solve issue #145.

hcoles commented 9 years ago

This would be great, but achieving it would be a complex challenging project in and of itself as it would require decompiling to mutated bytecode back to source.

The last time I looked at this the existing decompilers didn't generate great results and none of them could be used as a library.

mbj commented 9 years ago

@UrsMetz This is one of the reasons I decided to use the AST as transformable logic representation rather than using the bytecode. A mapping to source from AST and to executable bytecode from AST exists.

The trade off for choosing bytecode in the Java land was discussed between me and @hcoles multiple times in the past. I agree to his decision giving the circumstances in the JVM based world.

Out of curiosity: I'm writing a mutation tester for Haskell/GHC, here we have several AST representations one is desugared with syntax shugar expanded. I'm thinking about both the normal and the desugared representation to generate mutations, but not try to "reshugar" representations on the desguared AST.

I wounder if there is a language (not Java) where the combined bytecode and the mutated JVM bytecode could be uncompiled to. One that is still understandable by developers but does not require to "reshugar" to represent the bytecode in "shugarful" Java. Simular to haskells core. This language would allow to present nice "human consumable" diffs. Maybe something like C-- is a candidate? ( I never dealt directly with the JVM bytecode so this is a shot in the blue).

hcoles commented 9 years ago

Since I originally put pit together I think the java landscape has changed a bit, so programmatic in memory compilation of java is now an option. At some point I'd like to refactor things so that a source based mutator could be plugged into pit.

mbj commented 9 years ago

At some point I'd like to refactor things so that a source based mutator could be plugged into pit.

I was thinking in the past that we should be able to define mutation operators based on a shared meta definition. This way any JVM frontend could get the basic ones for free. Statement Deletion, Argument Popagation etc should be implementable in a gerneric source based way? In case the AST representation at least uses the same (but parameterized) datastructure.

UrsMetz commented 9 years ago

I had a quick look at the Procyon Decompiler and the decompilation results for mutants created by PIT. It looked promising :-). The results don't look exactly like the original source code (formatting and stuff like this. in front of instance variables are different) but with the examples I tried it was possible to find out what the mutator did.

UrsMetz commented 9 years ago

The challenging part I think would be to only display the part that was changed by the mutant (or at least the method where the change was made) as the decompiler just outputs the complete class. I might find the time to investigate this a bit further in the following weeks.

hcoles commented 7 years ago

I've finally had chance to take a look at this using procyon as @UrsMetz suggested.

Worryingly procyon doesn't look have been updated in a while.

http://www.benf.org/other/cfr/

Looks to be under active development, but is not easy to consume as it is closed source and not published to maven central.

I've put together a proof of concept in the decompilation branch

https://github.com/hcoles/pitest/tree/decompilation

This replaces the mutant description with a diff against the decompiled source.

There are a few issues

  1. Currently broken for Java 6 (would need to either drop java 6 support or move into a plugin)
  2. Adds a significant overhead. Decompilation will definitely need to be optional.
  3. Formatting is terrible.

Not sure the best way to go on point 3. I suspect the best solution would be change the html report so that the decompilation info appears in a dedicated section rather than replacing the existing description.

If we do stick with replacing the description it probably needs some markup added to format it. Doing this is complicated by some tech debt - markdown in the descriptions is currently automatically escaped.

If anyone with better html skills than mine would like to play with this they are welcome.