search

sixty-north / cosmic-ray

Mutation testing for Python
MIT License
556 stars 54 forks source link

cr-rate needs some attention #508

Closed abingham closed 2 years ago

abingham commented 3 years ago

cr-rate doesn't currently do its most basic task: report the survival rate. It has all sorts of stuff about estimates and confidence intervals injected into it, and - while I'm sure these are useful - they shouldn't be required by default.

tomato42 commented 2 years ago

For that exec would need to start executing the tests in random order again.

abingham commented 2 years ago

For that exec would need to start executing the tests in random order again.

I don't understand this. If I have a fully-executed session, the order or mutation seems irrelevant for calculating survival rate. For many cases, this final survival rate is what's important, and it's what cr-rate was initially created for.

tomato42 commented 2 years ago

Because when you're using it to actually estimate the survival rate from a partial run, you need to to have unbiased sample: selected at random from the population of all mutations. If the sample is just from one module, that may seriously under- or over-estimate the survival rate of the whole application, as that particular module can have significantly worse or better test coverage than other modules. If you just want the survival rate, you can simply look at last line of cr-report anyway.

Side note: if you run cr-rate and specify the --confidence 99.0 (or other valid), then it works just fine, the problem is the default value for the option parser

abingham commented 2 years ago

the problem is the default value for the option parser

Ah, right you are! This was much ado about nothing. I've updated the default argument value and fixed the tests in the process. Thanks for pointing that out.