Questions about custom Nimble matcher: hold()

[modocache]

I tried writing a Nimble matcher for PBT. This was before I learned about PBTAssert--it might be better to use that instead. For the time being, can you see any problems with this?

// Bridging header

#import <PBT/PBT.h>

import XCTest
import Nimble
// Unfortunately cannot `import PBT` here; must use bridging header

func hold<T: PBTGenerator>(numberOfTests: UInt = 100,
                                    seed: UInt32 = UInt32(time(nil))) -> MatcherFunc<T> {
  return MatcherFunc { actualExpression, failureMessage in
      let runner = PBTRunner() // Should this use `PBTRunner.sharedInstance()`?
      let result = runner.resultForNumberOfTests(numberOfTests,
          property: actualExpression.evaluate(), seed: seed)

      failureMessage.actualValue = "property"
      failureMessage.postfixMessage = "hold, but failed for \(result.smallestFailingValue)"

      return result.succeeded
  }
}

class HoldTests: XCTestCase {
  func testTwentyIsGreaterThanAllIntegers() {
    // Fails with "expected property to hold, but failed for 20"
    expect(PBTForAll(PBTInteger(), { value in
        return 20 > value as Int
    })).to(hold(numberOfTests: 800, seed: 8675309))
  }
}

Also, one thing that confused me about PBTInteger() was that it seemed to only generate numbers between -50 and 50. Is that intended (sorry if dumb question, still reading through source)? It was very unintuitive--as a property-based tester I think I'd expect far out numbers like -879,213, etc.

I like the idea of a Nimble matcher for PBT, but it should probably be packaged as a separate project. It can be distributed using CocoaPods or Git submodules & drag-and-drop into Xcode, like AshFurrow/Nimble-Snapshots..

[jeffh]

Yeah, it's an unintuitive implementation of the port of clojure's test.check (which also exhibits similar behavior). There is a maxSize parameter which allows you to tweak this. The side effect of setting it to a large value is when generating collections (which can be extremely expensive for large arrays of elements).

A short-term mitigation strategy is to up the default max size parameter. A longer way is to look into how to indicate when an integer generator is used for a collection (something that is slow), versus some thing less computationally expensive to generate (like numbers).

[jeffh]

Also, the hold matcher seems correct.

[modocache]

Also, the hold matcher seems correct.

Awesome, thanks! :+1: