Open simonetgordon opened 2 years ago
A health check error generally indicates a bug in the test suite. It means we are filtering too much in one of the strategies.
In that case, should I skip the test or disable the health check for now?
A health check error generally indicates a bug in the test suite. It means we are filtering too much in one of the strategies.
Yep, although when it comes to ivy
we have to make sure there's not an error on their end. We'd want to try running the extensive test suite for hypothesis.extra.array_api for ivy's PyTorch namespace. I had done something similiar before for ivy's JAX namespace https://github.com/data-apis/array-api-tests/issues/115#issuecomment-1120817175.
In that case, should I skip the test or disable the health check for now?
If you can figure using Hypothesis' internal suite as mentioned above, I'd try and explore that. But yeah otherwise I'd skip for now if you're running this on CI—I'd disable the health check only when exploring things locally.
To be sure, this particular health check error means that the test isn't actually being run at all, so you'd effectively need to treat this test as an XFAIL.
Is there a way to skip one of the special case functions for a particular method? e.g. skip test_binary
for add
? (also getting the heath check message for this)
Depends how, but yes. Specifying the test case externally (e.g. how we specify skips in CI), you'll need the unique test case id e.g. array_api_tests/test_special_cases.py::test_iop[__ifloordiv__(x1_i is -0 and x2_i > 0) -> -0]
.
I'm attempting to implement a similar set of functions to here down in the ivy conftest.py file. In principal, should this work? (currently not having any luck). Or is it necessary to have the skips.txt file within the array-api submodule directory itself as only the array api conftest.py can skip tests?
I'm attempting to implement a similar set of functions to here down in the ivy conftest.py file. In principal, should this work? (currently not having any luck). Or is it necessary to have the skips.txt file within the array-api submodule directory itself as only the array api conftest.py can skip tests?
Yeah, in principal... but specifying skips/xfails is a bit annoying to do with pytest, outside of them being literally coded in the test heh.
Or is it necessary to have the skips.txt file within the array-api submodule directory itself as only the array api conftest.py can skip tests?
I think you could have skips.txt
at the top level of this repo (not the array-api
submodule) for it to be registered with the sibling conftest.py
file.
Cheers, I think I have it working now!
I'm getting a lot of health check errors testing against the NumPy compat library https://github.com/data-apis/numpy-array-api-compat, specifically for floordiv
and mod
operators. Plain NumPy without the compat library fails because np.asarray
doesn't have the copy keyword. Is that a requirement for the iop tests to work?
Maybe we should add a plain NumPy build to the CI to check for test suite errors like this. Also, if you run the tests against the development NumPy with NPY_PROMOTION_STATE=weak
the type promotion tests all pass, so the number of test failures is much lower.
Plain NumPy without the compat library fails because
np.asarray
doesn't have the copy keyword. Is that a requirement for the iop tests to work?
Yep, on the in-place branch copy is used, as IIRC it's necessary to actually test the elements of resulting arrays.
Hi, I have encountered an hypothesis heath check error for the
__ifloordiv__
and other similar arithmetic operator methods intest_iop
when using tensorflow backend functions:What is the best way to handle this?