Can't consistently set number of examples

[reedrosenbluth]

I can't seem to set the number of examples in a test I'm writing. I have two almost identical tests, the only difference is the generator, which is also nearly identical, and the assertions in checkAssert. Setting the number of examples using either qt().withExamples(25) or System.setProperty("QT_EXAMPLES", "25") works in the first test, but does not seem to work in the second test, as adding a print statement indicates the test runs more than the set number of times. One edge case I've discovered is that if I set the number of examples to 1 it does in fact limit it to 1, but any other number seems to have no effect for this test.

Any idea how to fix this? Thanks.

[hcoles]

Thanks for the report, could you create a minimal project that reproduces the issue?

The number of examples used might vary if the generator finds a large number of duplicate, but I can't think of a scenario where it should generate more than the specified number of examples, only less.

[reedrosenbluth]

I've figured out when this is happening for me, and when it isn't.

Setting the number of examples works for most test cases. The test I've found where it doesn't work is a test that makes HTTP requests. Here's some "code" that resembles my test that isn't working properly:

qt().withExamples(10)
    .forAll(validRequests)
    .checkAssert(request -> {
        // Get some initial information about the state of system
        Response response = httpClient.makeRequest();
        Response secondResponse = httpClient.makeDifferentRequest();

        // Execute a transaction
        Response executeResponse = httpClient.executeTransaction(request);

        // Assert state is what we expect after the transaction
        assertThat(executeResponse.state(), is(StateEnum.WHAT_WE_EXPECT));
        assertThat(executeResponse.val(), is(response.val() + request.val())
        . . .

        // Revert the transaction to preserve the initial state
       Response executeResponse = httpClient.reverseTransaction(request);
});

If I comment out the executeTransaction() line and everything below it, the tests runs with the correct number of examples (10). If I leave it in, it keeps going way past 10.

Does this help? Any ideas why this would be happening?

[reedrosenbluth]

Sorry for the slow response btw, just got back from vacation and have been busy with some other stuff 😅

[reedrosenbluth]

@hcoles Have you gotten a chance to look at this?

[ProofOfPizza]

Hi I am experiencing the same issue. Both with withTestingTime and withExamples combined with testing api responses.

      qt().withExamples(20)
        .forAll(integers().all(), integers().all())
        .checkAssert((page, size) -> {
          String url = fetchurl 
          saveCallResults(getRequest(url, getTestContext().httpClient));
          int actual = getTestContext().requestResponse.getStatusLine().getStatusCode();
          System.out.println("print this line so e can count");
          assertThat(200, equalTo(httpCode)); // here withExamples works fine
          //assertThat(actual, equalTo(httpCode)); // using this one it runs indefinately (or at east for a looong time)
        });

A fix for this is much appreciated!

[manthanhd]

@reedrosenbluth @hcoles @ProofOfPizza I spent some time debugging this with a test codebase today.

It is not related to HTTP errors but when any kind of unchecked exception is thrown inside checkAssert the framework tries to shrink down to the shrink value (default is 0 in most numeric cases I think). Also, the shrinking mechanism works in cycles, each cycle being a call to the checkAssert method. By default, the number of cycles is 1 million or something really high in that order.

So if the original setting for examples in the test is say 20 and on the 19th attempt the test throws a RuntimeException, QT will start shrinking and will continue shrinking for 1 million (or equivalent count) times after that 19th attempt. Since there is no log to distinguish between the real attempt and a shrinking attempt, it looks like it is continuing indefinitely.

To fix this, just set the shrink cycles manually:

qt().withShrinkCycles(1).withExamples(10)...

Hope this helps!