search

jupyter / nbgrader

A system for assigning and grading notebooks
https://nbgrader.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
1.3k stars 317 forks source link

Intermittent rate limiting on docs build #1822

Open trevorcampbell opened 1 year ago

trevorcampbell commented 1 year ago

When building the docs (most recently the CI step below, which prompted me to file this issue, but other architectures cause the same issue sporadically):

Test python-doc / test_nbgrader (ubuntu-22.04, docs, 3.11) (pull_request)

I have noticed that the workflow is making many quick http requests and is occasionally getting rate limited by GitHub. For example, see https://github.com/jupyter/nbgrader/actions/runs/6005710823/job/16288942885?pr=1817 :

(user_guide/what_is_nbgrader: line  418) ok        https://noteable.edina.ac.uk/documentation/nbguide/
(user_guide/philosophy: line    5) ok        https://www.dropbox.com/s/olrx40rzzvk1v1i/Marr%20-%20The%20Philosophy%20and%20the%20Approach.pdf?dl=0
(contributor_guide/documentation: line   38) ok        https://www.sphinx-doc.org/en/master/
(       changelog: line   85) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Afranswel+updated%3A2023-03-28..2023-06-15&type=Issues
(       changelog: line   85) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Agithub-actions+updated%3A2023-03-28..2023-06-15&type=Issues
(       changelog: line   58) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Agithub-actions+updated%3A2023-06-15..2023-06-16&type=Issues
(       changelog: line   22) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Abrichet+updated%3A2023-07-18..2023-07-21&type=Issues
(       changelog: line  147) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Agithub-actions+updated%3A2022-08-22..2022-09-26&type=Issues
(       changelog: line   38) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Agithub-actions+updated%3A2023-06-16..2023-07-18&type=Issues
(       changelog: line  133) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Adependabot+updated%3A2022-09-27..2023-03-28&type=Issues
(       changelog: line   38) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Abrichet+updated%3A2023-06-16..2023-07-18&type=Issues
(       changelog: line   22) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Agithub-actions+updated%3A2023-07-18..2023-07-21&type=Issues
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajhamrick+updated%3A2022-05-07..2022-06-28&type=Issues | sleeping...
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajhamrick+updated%3A2022-05-07..2022-08-22&type=Issues | sleeping...
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2022-05-07..2022-08-22&type=Issues | sleeping...
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2022-05-07..2022-06-28&type=Issues | sleeping...
(       changelog: line  133) broken    https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2022-09-27..2023-03-28&type=Issues - 429 Client Error: Too Many Requests for url: https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2022-09-27..2023-03-28&type=Issues
(       changelog: line  133) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Agithub-actions+updated%3A2022-09-27..2023-03-28&type=Issues
(       changelog: line  267) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2022-05-07..2022-06-28&type=Issues
(       changelog: line  196) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajhamrick+updated%3A2022-05-07..2022-08-22&type=Issues
(       changelog: line  267) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajhamrick+updated%3A2022-05-07..2022-06-28&type=Issues
(       changelog: line  196) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2022-05-07..2022-08-22&type=Issues
(       changelog: line   38) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2023-06-16..2023-07-18&type=Issues
(       changelog: line  133) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ameeseeksmachine+updated%3A2022-09-27..2023-03-28&type=Issues
(       changelog: line  267) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Aminrk+updated%3A2022-05-07..2022-06-28&type=Issues
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Aperllaghu+updated%3A2022-05-07..2022-06-28&type=Issues | sleeping...
(       changelog: line  196) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Aminrk+updated%3A2022-05-07..2022-08-22&type=Issues
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Aperllaghu+updated%3A2022-05-07..2022-08-22&type=Issues | sleeping...
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Aperllaghu+updated%3A2022-09-27..2023-03-28&type=Issues | sleeping...
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Arkdarst+updated%3A2022-05-07..2022-08-22&type=Issues | sleeping...
Error: The action has timed out.

This causes CI to look like it's failing, even when nothing is wrong with the build itself. I believe the fix may just be to slow down these requests a bit, but not sure where that fix should be made and what an appropriate rate is.

The Error: The action has timed out. may actually be GitHub killing the action. It looks as though the sphinx-build -b linkcheck command in the docs Makefile is at least aware of rate-limiting and maybe (?) is exponentially backing off ( see https://github.com/sphinx-doc/sphinx/issues/7388 ) , but perhaps GitHub sees the action sleeping and decides to kill it.

Another potentially useful observation: it seems that the failure occurs shortly after 429 Client Error: Too Many Requests for url: ... -- so perhaps it is indeed that sphinx-build isn't backing off quickly enough...

brichet commented 1 year ago

Thanks @trevorcampbell for opening this issue. The timeout for the documentation is set to 15 minutes in the github workflow: https://github.com/jupyter/nbgrader/blob/9bc0a14531378fd950fd7c4dfddde471d194bfb1/.github/workflows/test-docs-python.yml#L51

Yesterday, a few changes concerning the documentation were merged (https://github.com/jupyter/nbgrader/pull/1819). I don't know if this improves the duration but the test following the changes took less than 12 minutes: https://github.com/jupyter/nbgrader/actions/runs/6004868762/job/16286405021#logs

trevorcampbell commented 1 year ago

@brichet Since you merged #1819 I haven't seen the issue again. But it was somewhat intermittent before, so hard to tell if it's actually gone.

trevorcampbell commented 1 year ago

Hah. 10 minutes after I wrote that, it happened again.

https://github.com/jupyter/nbgrader/actions/runs/6018862753/job/16327800386?pr=1817