search

mabdi / small-amp

Test Amplification for Pharo
MIT License
5 stars 4 forks source link

Negative mutation score improvement #78

Open mabdi opened 4 years ago

mabdi commented 4 years ago

In the evaluation, some classes show negative improvement.

Technically, it can not happen. Because in the worst case, we don't have any new test method to suggest, and mutation score shouldn't be changed.

TODO: check the reason. (flaky test?)

mabdi commented 4 years ago

We see flakyness in generated tests:

(code from: project: zinc class ZnEntityReaderTest)

testReadStreaming_amp_L15
    <madeBySmallAmp>
    | data headers reader entity aCollection aCollection0 |
    aCollection0 := 1 to: 32.
    aCollection := aCollection0 collect: [ :each | (-1 to: 9) atRandom ].
    self should: [ data := aCollection asByteArray ] raise: Error

when at least one -1 presents in the aCollection we will have an error. if no -1 is presented (probability: (10/11)^32 ~= 0.047) we will have an error.