search

mbj / mutant

Automated code reviews via mutation testing - semantic code coverage.
Other
1.94k stars 151 forks source link

feat: todo file #1435

Open devinburnette opened 5 months ago

devinburnette commented 5 months ago

For teams that use --since in their CI pipelines, but also block merges on successful builds, it's kind of frustrating to do a large chore, like a rubocop adjustment or deleting dead code or something like that and have mutant complain about these older classes that lack 100% mutant coverage.

I think it would be nice if mutant had a way to dump all the mutations it found to a .mutant_todo.yml similar to rubocop for situations like these. Let me know what you think.

mbj commented 5 months ago

@devinburnette Overall: Yes, in detail: We should not replicate rubocop todo file semantics as these assume you can do a pass over the entire code base in a reasonable time to generate it.

Most of the code bases mutant is used at are simply to big to get a full pass without --since. Instead we should have an "at commit level" signal for mutant to simply skip mutation analysis for a specific commit. I think this would be superior, right?

devinburnette commented 5 months ago

generating a todo file is usually a one time operation run manually and not in CI, so I don't think there's an expectation that it completes in a reasonable time.. I think it's also a good way for teams to take an incremental approach at improving their code pre-mutant..

I don't quite think that skipping a specific commit solves for what I'm getting at... While onboarding mutant, I want a way to say ignore everything before this point.. (which the --since flag does well).. but then additionally I want to say going forward if I touch a previously ignored subject, keep ignoring it until I've checked it off the todo list..

i think for some teams it's too much to assume the burden of killing mutations for changed subjects that existed pre-mutant.. although, I realize this defeats the purpose of mutant because if you do this you can't totally be sure the code you changed does what it's supposed to do, so it should be opt-in with a big warning saying so.

mbj commented 5 months ago

I'm happy to support a mode where mutant generates an ignore list, if we put a warning in about the runtime of that generator.

But: I wounder if we can also improve --since a bit to actually do an AST diff, not a source diff. This way 99% of "rubocop fixes" would not trigger --since as these are often AST neutral?

EDIT: And for the cases its "not AST neutral" you actually wanna verify coverage, right?

EDIT 2: When I say ast neutral I mean AST without source locations.