search

data-apis / array-api-tests

Test suite for the PyData Array APIs standard
https://data-apis.org/array-api-tests/
MIT License
65 stars 42 forks source link

Skipped flaky tests #300

Open asmeurer opened 3 days ago

asmeurer commented 3 days ago

Several tests are completely skipped right now because they are "flaky".

This is a pretty high priority issue because these functions are effectively completely untested, even though they appear to be tested.

Tests should be written in such a way that they aren't flaky, for instance, by using high numerical tolerances (or if necessary, avoiding values testing entirely).

Note that health checks for timeouts should just be skipped, and health checks for filtering too much should be fixed by fixing the strategy.

ev-br commented 23 hours ago

Looking at test_std, https://github.com/data-apis/array-api-tests/blob/master/array_api_tests/test_statistical_functions.py#L262, it does not seem to attempt any value testing. Then what is flaky, assert_dtype or assert_keepdimable_shape?