search

data-apis / array-api-tests

Test suite for the PyData Array APIs standard
https://data-apis.org/array-api-tests/
MIT License
63 stars 39 forks source link

Reset the max-examples default back to 100 #254

Closed asmeurer closed 3 months ago

asmeurer commented 4 months ago

The tests run fast enough now that there's no need to lower this to 20.

See the discussion at #246.

honno commented 4 months ago

Could we just change our own workflow to have --hypothesis-max-examples=100, instead of the default? As defo for something like JAX this makes it basically unusable on default runs :sweat_smile:

asmeurer commented 4 months ago

JAX is the exception, not us. If JAX is slower than most libraries, they should manually lower their max examples. Library defaults should match the common case, not the least common denominator.

asmeurer commented 3 months ago

I also changed the unvectorized max examples to scale linearly with max-examples instead of logarithmically. Otherwise, it's very difficult to actually run a lot of examples for unvectorized tests. We can adjust the scaling factor if 1/10 is not small enough by default.