Closed dhimmel closed 4 years ago
Doing HEAD rather than GET requests in https://github.com/weecology/ogrants/pull/108/commits/2c2c760da55862fa7515604a750ba39dbbb32ac3 reduced the link validation stage from 170 to 52 seconds. There is some possibility that websites will return proper HEAD responses but fail with GET although AFAIK this is rare.
https://github.com/weecology/ogrants/pull/108/commits/d4134f42791117c889f05506b356980f17c0e3d4 makes it so only grants that are changed in a pull request / push commit are tested. This speeds things up and makes it so test failure is specific to the changeset.
It should test all links if the Travis build is a cron job. Therefore, if accepting this change, I'd recommend enabling weekly cron jobs in the Travis CI settings. This way, every week, all links will be validated. But PRs will only test the links that are changed/added.
https://github.com/weecology/ogrants/pull/108/commits/1df3948ef066cd39db687692fe3704b18aa84cca adds the ability to use an environment variable to test all (useful for local development):
OGRANTS_TEST_ALL=1 python tests/validate_links.py
This is awesome! Thanks @dhimmel! Cron job set.
@ethanwhite ready on my end. Consider squash merging this since there are many debugging / bugfix commits.