pragdave / quixir

Property-based testing for Elixir
266 stars 11 forks source link

Shrinker is converging on the wrong value #14

Open lboekhorst opened 7 years ago

lboekhorst commented 7 years ago

First off, my environment is as follows:

I am testing that a field on my struct is only valid when it is a list where all values are between one and seven:

test "changeset is valid when list values between one and seven" do
  ptest [list: list(int(min: 1, max: 8))] do
    params = string_params_for(:my_struct, list: list)
    changeset = MyStruct.changeset(%MyStruct{}, params)
    assert changeset.valid?
  end
end

I purposely set max to eight to test that it fails and provides the proper failing list. It seems it doesn't however. The first example this test case generates is:

%{list: [1, 1, 6, 6, 8, 3, 6, 2, 5, 2, 3, 7, 6, 2, 3, 5, 7, 1, 8,
   5, 2, 6, 3, 8, 5, 6, 7, 3, 5, 6, 6, 2, 2, 8, 4, 2, 7, 1, 5, 3, 3, 3, 4, 1, 2,
   8, 1, 5, 6, 1, ...]}

This list includes eights so it properly concludes the changeset is not valid. Then it will try to shrink it down to the smallest possible failing test case:

[6, 3, 6, 8, 8, 3, 6, 3, 2, 1, 5]
false
[3, 6, 8, 8, 3, 6, 3, 2, 1, 5]
false
[6, 8, 8, 3, 6, 3, 2, 1, 5]
false
[8, 8, 3, 6, 3, 2, 1, 5]
false
[8, 3, 6, 3, 2, 1, 5]
false
[3, 6, 3, 2, 1, 5]
true

What you are looking at is the result of the shrinked list, and the result from changeset.valid? given that list. Up until the final example the list is still invalid since it contains an eight. The final list no longer includes an eight and thus is valid. However, at this point, quixir will stop shrinking and output the following:

Expected truthy, got false
when: list = [3, 6, 3, 2, 1, 5]

This is incorrect, this is the first test case that is passing for a valid reason. Am I overlooking something here? How can I get it to properly report the failing test case? I would also be ok with an option to disable shrinking (since this seems to be the cause) altogether as a quick workaround.

ghost commented 7 years ago

Works for me with IEx 1.5.0:

root@localhost(@)> iex --version
Erlang/OTP 20 [erts-9.0] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:10] [hipe] [kernel-poll:false]

IEx 1.5.0
Mon Oct 16 09:50:24
~/quixir
root@localhost(@)> mix test
.............

Finished in 0.4 seconds
13 tests, 0 failures

Randomized with seed 134644
Mon Oct 16 09:50:38
~/quixir

Please try upgrading elixir to the latest stable version.