Add a type for complex input generation that performs exploratory testing

[SeanROlszewski]

Description

In order to support testing patterns where randomized input generation is used to explore a program's state space (such as fuzz, differential, property-based, or mutation testing) we need to add some affordance to developers to specify the following:

1) The next value to pass into a @Test 2) A way to receive the prior value and test result as a result of executing @Test 3) A way to receive additional information, such as changes in code coverage, from the last @Test 4) A way to stop input generation based on some heuristic or predicate.

Having these 4 items would be the essential set of primitives to really streamline the creation of tools like QuickCheck, libFuzzer, etc. with Swift Testing.

Perhaps this could take the form of a protocol, potentially named TestCaseInputGenerator, that has the following shape:

protocol TestCaseInputGenerator {
    associatedtype AdditionalContext
    func shouldStop<InputType>(given priorRun: (InputType, TestResult)?, additionalContext: AdditionalContext?) -> Bool
    func nextValue<InputType>(given priorRun: (InputType, TestResult)?, additionalContext: AdditionalContext?) -> InputType
}

AdditionalContext can encapsulate data of interest to the test; previously tried values, their results, and other data like code coverage, crashes, etc.

I'm filing this issue rather quickly, in between tasks, so apologizes in advance for oversights on the type signatures and compilability of the proposed API. :)

[samritchie]

There doesn’t seem to be an awful lot of interest in this feature - let me know what I can do to help? I’ve been working (very slowly) on a hedgehog port. The types of things I’d be looking for are:

• Top requirement is suppressing expectation failures - generally for property based tests you may have hundreds of failures but you only want the minimal failing case reported; the rest is noise
• It’s not clear how/if test runs can be parallel. Ideally the root level tests would run in parallel, but shrinks should be tested sequentially
• Unsure whether variadic inputs would be supported? Vanilla parameterised tests appear to only support up to two arguments, but this restriction may not be as applicable to property tests which don’t attempt all combinations
• Reporting output: I assume the relevant info would need to be shoehorned into the existing issue reporting process - it looks like it should work, but for reference, the typical output is:
• Passed: number of (root) tests
• Failed: number of tests, number of shrinks, input value(s), failure description (ie expectation failure narrative), seed/path info to reproduce the test. Inputs & failures should be catered for, the rest may need to go to stdout?

Most libraries also have a verbose option that logs out inputs for all tests run, this could potentially be recorded the same way as the current parameterised test runs, although the number of sub-tests is potentially large.

An alternative approach may be some sort of test interceptor or embedded runner that can receive, filter, modify & generate test events/issues within its test context - this is the workaround I ended up using for XCTest support. I don’t think this would be as well-integrated into swift-testing overall but would be more flexible and may be simpler to implement.