Add CI test for docs build of selected Scientific Python projects

[melissawm]

To avoid breakage and unexpected problems when a release is made, we could add a CI job that builds the docs pages for a list of selected projects and verifies that the output is as expected. Ideally, it would run as a cron job (and not on every CI run).

[melissawm]

The following projects seem like reasonable candidates:

• https://scipy.github.io/devdocs/index.html -> This is a very time-consuming build, but can be useful precisely for benchmarking purposes.
• https://docs.jupyter.org/en/latest/
• https://awkward-array.org/doc/main/
• https://pytest-mpl.readthedocs.io/en/stable/

[12rambau]

What would be the "expected output" ? It's very difficult to get CI to check for how things render on our own website already.

[trallard]

I think there are two things that could be tested

1. That the build process of third party docs is successful - this can be easily checked in CI and mostly helps us catch any unintentional breaking changes introduced by us or some of our dependencies.
2. Visually inspecting the build - this is definitely hard from a CI only POV and unless we have visual regression tests (which would be a great amount of work on our end to write and maintain as we continue to work on the theme) there is no way to do but through a manual check. So IMHO this should be out of our scope.

So we should only focus on 1. for the time being

[melissawm]

For completeness, it may be also helpful to have https://github.com/executablebooks/sphinx-book-theme

[drammock]

For completeness, it may be also helpful to have https://github.com/executablebooks/sphinx-book-theme

yep. We recently broke something for them (https://github.com/pydata/pydata-sphinx-theme/issues/1875#issuecomment-2162995192) so definitely makes sense to include