search

sphinx-doc / sphinx

The Sphinx documentation generator
https://www.sphinx-doc.org/
Other
6.43k stars 2.1k forks source link

efficient 2-stage parallel builder? #7891

Open dimpase opened 4 years ago

dimpase commented 4 years ago

Due to slowness of Sphinx in buidling of the reference manual of SageMath several years ago we implemented a paralell 2-stage customised Sphinx builder, details may be found in https://trac.sagemath.org/ticket/20080

In a nutshell, it runs Sphinx once to generate intersphinx "object.inv" inventory files and pickle files; these are used in a parallelised 2nd run, where independent builders process subcollections of documents independently, using intersphinx extension to make sure cross-references are maintained; finally, the global index is built.

This gives a many-fold speedup on multi-core CPUs (to know by how much exactly, one would need to create a "standard" Sphinx setup for SageMath, and time it against our builder - but certainly if you have, say, 8 CPUs cores available, the speedup is about 5 to 6-fold, if compared with just using 1 CPU).

As Sphinx evolves, our setup gets harder to maintain (it does not help that several people who originally implemented it drifted away from the project), so we would like to streamline and upgrade our setup, using as much as is available from Sphinx upstream. Perhaps upstream might be interested in having a similar parallel builder based on this idea, then we could just use it, and do not maintain our builder.

tk0miya commented 4 years ago

A while ago, I tried to build the document of sagemath via the original sphinx-build. But I can't find the way to build it. I need to know the structure of the document. It seems https://github.com/sagemath/sage/tree/develop/src/doc contains many sphinx project. @dimpase Is there any document or article for the structure of docs? I'd like to measure how slow the standard build at first.

dimpase commented 4 years ago

By far the biggest document there is reference. The build time of it dominates the rest. Thus I think it would be the interesting enough to measure on this document alone.

Most of it consists of docstrings in Sagemath Python/Cython library, i.e. in https://github.com/sagemath/sage/tree/develop/src/sage/

dimpase commented 4 years ago

more precisely, the document I mean is here: https://github.com/sagemath/sage/tree/develop/src/doc/en/reference

tk0miya commented 4 years ago

Now I'm trying to create an environment to build the document... it seems I need to install many dependencies to build the docs...

dimpase commented 4 years ago

This is correct - you basically might need a working copy of Sagemath. Do tell me if you have a problem with it; basically, building Sage is well-supported on stable versions of Linux, such as Debian, Ubuntu, Fedora. It's OK on macOS, too. Windows is very painful though.

One way or another, it might be quicker to get Sagemath on conda.

tk0miya commented 4 years ago

It would be very helpful to me if there are well-prepared docker image (or Dockefile) to build the environment. I don't have experience of conda. So this is very tough for me.

dimpase commented 4 years ago

Please try https://hub.docker.com/r/sagemath/sagemath/

Hopefully @mkoeppe and @saraedum can tell you more about Sagemath on docker, if needed.

tk0miya commented 4 years ago

Thank you for the info. I'll try it this weekend.

dimpase commented 3 years ago

I wonder if you had a chance to try this out. Our doc builder keeps getting us troubles.

tk0miya commented 3 years ago

Sorry for the late response. I just tried the docker image now. But I still don't find the way to build the doc of sagemath yet. It's very helpful if somebody let me know the way to build the document of sagemath on my local.

dimpase commented 3 years ago

@mkoeppe is there a way to quickly set up a docbuilding environment for Sage in docker?