search

stryker-mutator / mutation-testing-elements

🗃 A schema for mutation testing results with the web components to visualise it 📊
Apache License 2.0
28 stars 19 forks source link

Commented out code marked as mutated in html report #1931

Open Bforslund opened 2 years ago

Bforslund commented 2 years ago

Describe the bug It seems that code that is commented out when part of a block mutation is marked as mutated code. I think the report should ignore lines that are comments from the visualisation.

Desktop (please complete the following information):

Additional context image

dupdob commented 2 years ago

Stryker uses Roslyn to parse the source and then mutate the resulting syntax tree. Actually it cannot mutate commented out code, because Roslyn reports it as comments, which are not mutated.

What you see here is the mutation for the whole code block, it appears in green below. Block mutations do not preserve comments.

rouke-broersma commented 2 years ago

We should move this to mutation testing elements. It can be confusing to see a comment marked as being a part of a mutation and users might interpret this as commented code being mutated.

hugo-vrijswijk commented 2 years ago

I'm not sure what would be a good solution here. I'd be happy to hear suggestions.

IMO marking the whole block as NoCoverage makes sense, because the entire block has no coverage. It's different from e.g. just the await statement being mutated

Also, on a technical level I think this could be fairly complex to implement and make exclusions for

dupdob commented 2 years ago

Side comment: not sure why this mutation happened since I have added a logic preventing mutating bloc with a single statement, as it doubles with the statement mutator. Technically, the comment IS mutated, since the mutation bears no comment at all.

Is this a problem only because there is code in the comment ? What about code in a string ? Would people expect it to be mutated as well.

This problem exists (as well as other) because users have strong assumptions about how Stryker works (by altering source code text).

Just thinking out loud here, not saying nothing should be done, but I tend to think somehow excluding the comment is not the best way to address this concern.

rouke-broersma commented 2 years ago

Perhaps we could add an optional property to the schema for lines that are included for visual reasons only. It would then be up to the tool to decide which lines if any are not mutated code.

rouke-broersma commented 2 years ago

Is this a problem only because there is code in the comment ? What about code in a string ? Would people expect it to be mutated as well.

I think it is only a problem because there is code in a comment. You're refactoring, you're commenting out some code you want to remove etc yet in the mutation report or appears as if the code was active.