More roboust remote testing - Githubissues

astropy / astroquery

Functions and classes to access online data resources. Maintainers: @keflavich and @bsipocz and @ceb8

http://astroquery.readthedocs.org/en/latest/

BSD 3-Clause "New" or "Revised" License

706 stars 399 forks source link

More roboust remote testing #2113

Open bsipocz opened 3 years ago

bsipocz commented 3 years ago

The example in #2112 unearthed the issue that our remote-testing protocol is not sufficient enough, we could have catch the bug if the CI matrix was running remote data tests for the old dependency combination, too.

There are a few options we could do, and I raise this issue with the infrastructure team to see what is the best way forward:

set up a system to test on module level, and run remote data tests in PRs that are either triggered by:
- label
- automatically (but for the module only)

The main issue with remote testing is still that running all of them:

takes a lot of time (1+ hour)
stresses remote services even when changes is clearly unrelated
there could be a lot of false positives, we don't want to have a constant failing CI just because one or two services have downtimes or issues

bsipocz commented 3 years ago

We could even cut up the remote cron tests along the line of archives, e.g. run all ESA or all IPAC or all ALMA, and what's remaining, etc together to make it easier to set up a notification system.