typelift / SwiftCheck

QuickCheck for Swift
MIT License
1.42k stars 106 forks source link

String generator breaks if test not always true #289

Closed starbase527 closed 5 years ago

starbase527 commented 5 years ago

Version

077c096c3ddfc38db223ac8e525ad16ffb987138

Environment

macOS 10.14.5, Xcode 10.2.1, Swift 5.0.1, SwiftCheck 0.12.0

Description

When forAll sometimes evaluates false using the generator in the code below, things get weird.

Steps To Reproduce

import SwiftCheck

let metreGen = Gen<String>.fromElements(of: ["m", "metre", "metres"])

property("works fine") <- forAll(metreGen) { (unit: String) in
    print(unit)
    return true
}

property("doesn't work") <- forAll(metreGen) { (unit: String) in
    print(unit)
    return false
}

The first property produces expected output of the form

metre
metres
metres
m
metre
m
m
metre
metre
metre
metres
m
m
metres
metres
metre
m
metres
metre
metre
m
metre
metre
metres
metre
metre
metres
metre

etc.

The second seems to produce a single result as expected, then things get weird. It also seems to run indefinitely, sometimes terminating with a error: Execution was interrupted, reason: EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0). Output:

metres
*** Failed! es
tres
mees
metr
etres
mtres
meres
metes
metrs
metre
1etres
aetres
2etres
 etres
Cetres
cetres
betres
Aetres
Betres

etres
3etres
mctres
m
tres
m2tres
matres
m tres
m1tres
mCtres
m3tres

etc.

I'm only just learning Swift, so I'm struggling to find any leads in the source code at the moment.

sebastiangrail commented 5 years ago

This is expected behaviour: The shrinker is not aware of the generator. You are using a custom generator, but once your forAll fails, SwiftCheck tries to find a minimal failing example by shrinking the failing test case. SwiftCheck is using the default shrinker for the type (see https://github.com/typelift/SwiftCheck/blob/master/Sources/SwiftCheck/Arbitrary.swift#L316) unless you specify a custom shrinker with forAllShrink. If you don't want SwiftCheck to shrink your failing test cases, you can also use forAllNoShrink, but you'll lose a lot of the power of property based testing.

See also this question on shrinking behaviour

I couldn't reproduce the crash with the code you posted above.

CodaFi commented 5 years ago

I'm going to close this because there hasn't been an update. Thank you @sebastiangrail for such a quick and thorough response to this issue. Please feel free to reopen it if I've made a mistake.