validate-links: re-enable link checker

w3c / spec-prod

GitHub Action to build ReSpec/Bikeshed specs, validate output and publish to GitHub pages or W3C

https://w3c.github.io/spec-prod/

MIT License

34 stars 21 forks source link

validate-links: re-enable link checker #40

Open sidvishnoi opened 3 years ago

sidvishnoi commented 3 years ago

Link checker is disabled due to bugs that needs to be fixed:

[ ] some requests get rate limited (HTTP 429) e.g. github.com.
[ ] skip certain URLs, e.g. the to-be published /TR URL (HTTP 404).

johanneswilm commented 1 month ago

Hey,

we are seeing the issue of a 429 from github.com due to link-checker being reenabled [1].

@marcoscaceres I noticed github.com was not added to the list of URLs to be ignored when reenabling the link-checker. Could it possibly be added or is there some other way to disable this link-checking for github URLs?

[1] https://github.com/w3c/input-events/actions/runs/11405946266/job/31738562342#step:3:616

could not fetch external page: Error: Too Many Requests (Code: 429) from index.html to https://github.com/w3c/input-events/commits/

marcoscaceres commented 1 month ago

Yeah, that's weird (that it is failing for that reason)... it shouldn't be in the list of ignored URLs tho.

Can you try running the checker again? I've not seen this be an issue anywhere else.

johanneswilm commented 1 month ago

@marcoscaceres I just reran the same check and the error message is the same. See https://github.com/w3c/input-events/actions/runs/11405946266/job/31956123357 .

As the issue seems to be related to github throwing the 429 on multiple consecutive requests for that URL, I wonder if a solution would be to either lower the number of links going to github within the document (do we have more links than other specs?) or, alternatively, that we add a small sleep timer between each request to the same domain in the link checker.