search

weecology / ogrants

Open grants list
https://www.ogrants.org/
Other
125 stars 46 forks source link

skip link validation and test refactor #108

Closed dhimmel closed 4 years ago

dhimmel commented 4 years ago

@ethanwhite ready on my end. Consider squash merging this since there are many debugging / bugfix commits.

dhimmel commented 4 years ago

Doing HEAD rather than GET requests in https://github.com/weecology/ogrants/pull/108/commits/2c2c760da55862fa7515604a750ba39dbbb32ac3 reduced the link validation stage from 170 to 52 seconds. There is some possibility that websites will return proper HEAD responses but fail with GET although AFAIK this is rare.

dhimmel commented 4 years ago

https://github.com/weecology/ogrants/pull/108/commits/d4134f42791117c889f05506b356980f17c0e3d4 makes it so only grants that are changed in a pull request / push commit are tested. This speeds things up and makes it so test failure is specific to the changeset.

It should test all links if the Travis build is a cron job. Therefore, if accepting this change, I'd recommend enabling weekly cron jobs in the Travis CI settings. This way, every week, all links will be validated. But PRs will only test the links that are changed/added.

https://github.com/weecology/ogrants/pull/108/commits/1df3948ef066cd39db687692fe3704b18aa84cca adds the ability to use an environment variable to test all (useful for local development):

OGRANTS_TEST_ALL=1 python tests/validate_links.py
ethanwhite commented 4 years ago

This is awesome! Thanks @dhimmel! Cron job set.