Fuzzer classification and coverage

[Janiczek]

Fixes #94

• [x] docs
• [x] statistical test, to reduce flakiness
• [x] Fuzz.labelExamples : Int -> List ( String, a -> Bool ) -> Fuzzer a -> List ( List String, Maybe a )

---

• [x] Test.FuzzOptions now contains coverage : Coverage a

---

• [x] Test.Coverage a = Test.noCoverage | Test.reportCoverage labels | Test.expectCoverage labels
• [x] Test.noCoverage : Coverage a
• [x] Test.reportCoverage : List ( String, a -> Bool ) -> Coverage a
  • [x] report after running the test (Runner.String)
  • [x] coordinate with elm-test and elm-test-rs CLIs
  • [x] write a PR for elm-test-rs's elm-test-runner - https://github.com/mpizenberg/elm-test-runner/pull/13 and https://github.com/mpizenberg/elm-test-rs/pull/118
  • [x] write a PR for node-test-runner - https://github.com/rtfeldman/node-test-runner/pull/614
  • [x] off by one error (total number of coverage ticks for {runs = 100} is 99)
• [x] Test.expectCoverage : List ( ExpectedCoverage, String, a -> Bool ) -> Coverage a
  • [x] silent if coverage is satisfied
  • [x] fail the test if coverage is not satisfied

---

• [x] Test.ExpectedCoverage = Test.zero | Test.atLeast n | Test.moreThan n
• [x] Test.zero : ExpectedCoverage
  • [x] fail if 1+ values fell into the given category
• [x] Test.atLeast : Float -> ExpectedCoverage
  • [x] fail if < the expected number of values fell into the given category
• [x] Test.moreThanZero : ExpectedCoverage
  • [x] fail if 0 values fell into the given category

Fuzz.labelExamples

Test.reportCoverage

Test.expectCoverage

zero

more than zero

at least

[Janiczek]

@gampleman raises a good question in #94 about using Expect.classify or Expect.cover (modulo names) instead of having this in FuzzOptions (thus living in Test). Still unsure about it. I'll first implement the statistical test for reducing flakiness which we need no matter where this lives 🙃

[mpizenberg]

I'm not well versed on this subject and can't check it before mid next week. So if you feel confident about one way of doing it, I won't be against. If you prefer waiting for a more detailed answer from me, I'll need some time.

On Sat, Sep 10, 2022, 18:54 Simon Lydell @.***> wrote:

@.**** commented on this pull request.

In src/Test/Coverage/Internal.elm https://github.com/elm-explorations/test/pull/187#discussion_r967677090:

+{-| Coverage checks will be allowed to give a false positive once in certainty

+runs (that is, with probability 1/certainty).

+

+The current number was taken from Haskell QuickCheck. Their documentation says

+(quote):

+

+> If you are using 'checkCoverage' as part of a test suite, you should

+> be careful not to set @certainty@ too low. If you want, say, a 1% chance

+> of a false positive during a project's lifetime, then @certainty@ should

+> be set to at least @100 * m * n@, where @m@ is the number of uses of

+> 'cover' in the test suite, and @n@ is the number of times you expect the

+> test suite to be run during the project's lifetime. The default value

+> is chosen to be big enough for most projects.

+

+In the future we might want to make it configurable, as it is a knob to make

+tests finish faster at the expense of larger probability of false positives.

I have honestly not followed this closely enough so I don’t really know. But we already do some regex on the JS so adding some more for this if needed sounds good to me!

— Reply to this email directly, view it on GitHub https://github.com/elm-explorations/test/pull/187#discussion_r967677090, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAWFOCPDC5AZUB4QRJ2VBVLV5S4NVANCNFSM54YWLBCQ . You are receiving this because you were mentioned.Message ID: @.***>

[Janiczek]

Pulling the plug on this as we're nearing the release! There's still a week or so left, please shout if you find an issue or have a question or there is something to reconsider.