search

muter-mutation-testing / muter

🔎 Automated mutation testing for Swift 🕳️
MIT License
498 stars 39 forks source link

Muter stops early when it detects a certain number of consecutive build errors #40

Closed SeanROlszewski closed 5 years ago

rakaramos commented 5 years ago

hey @SeanROlszewski, what do you have in mind for this feature?

If I got it right, it would be:

  1. An optional, configurable JSON property.
  2. When muter detects that the threshold has been reached (consecutively!), it quits (reporting with all the results so far?)

What do you think?

Thanks!

SeanROlszewski commented 5 years ago

@rakaramos we're thinking similarly! For the purposes of this issue, I would hold off on adding JSON property - I want to see through an example codebase how painful it is to not be able to configure the error threshold, but I can imagine this being added in the future.

I think it's great if we can get it to report the results so far. If you run into issues getting that going, then it's acceptable for it to bail out early and just report that something anomalous happened.

The spirit of the issue is to have Muter save the user time by stopping if it detects something anomalous/weird with the results it's getting. Let's say 5 consecutive build failures is anomalous for now.

When it does bail out, it should print this message:

Muter noticed the last 5 attempts to apply a mutation operator resulted in a build error within your code base. 

This is considered unlikely and abnormal. If you can reproduce this, please consider filing an issue at
https://github.com/SeanROlszewski/muter/issues/
SeanROlszewski commented 5 years ago

Also, I'll steer you towards adding a test in mutationTestingSpec.swift. I'd imagine you would add another context block beneath "when the baseline test run passes". I'm happy to pair with you on what a test for this could look like :)