mgorny
commented
7 years ago Well, I've tried to give it a shot… and I can't find a sane way of doing it with the current design.
The big problem is, the use of pickles is done quite inconsistently. The pickled environment is first read in Sphinx._init_env() which is called by Sphinx.__init__(), i.e. pretty early. However, it is not updated until Builder.build(), and there it is pickled afterwards. So to get the results I need, I would have to lock the environment from Sphinx._init_env() to Builder.build().
To do that, I would have to keep the environment file open during that period. That's quite doable — I was thinking of opening it directly in Sphinx._init_env(), and keeping the open file in BuildEnvironment. This would have the extra advantage of not having to repeatedly pass the path around. In this case, Sphinx._init_env() would open the file r+ (i.e. open R/W or create), and lock it. BuildEnvironment.frompickle() would just read the open file (if non-empty), and BuildEnvironment.topickle() would rewind and write it. Then it could be unlocked and closed.
The problem with all that is that Python doesn't do RAII properly, so there's no clean way of ensuring that the file will be closed on exception without redesigning how Sphinx is used. Probably the whole class would have to be turned into context manager, and used via with like files are usually opened. Otherwise, I'm afraid that some corner case may result in the file being kept open (and locked), and deadlocking some other process.
tk0miya
zenhack
ehabkost
Projects using Sphinx (and Makefiles) often define multiple targets for different doc builders. For example, the LLVM project defines a target for manpages and for HTML docs. Since those targets are independent, make is allowed to run them in parallel (and does that). Since they use the same source tree, they use the same doctree root as well.
Lately, I've noticed that the two parallel Sphinx instances 'fight' over the doctree. Basically, it seems that they both ignore each other and create the doctrees independently, i.e. the same files are first written by one Sphinx instance, and afterwards overwritten by the second. While normally this doesn't cause problems (except for being awfully inefficient), I've actually seen a race condition causing major failure (sorry, don't have any logs for it). Basically, one Sphinx instance has opened one of the files for reading while the other was writing it.
We've attempted to find a good solution on the build system side but could fine none satisfactory. If we switched to separate doctrees, that would be a waste of a good caching opportunity. Getting two independent targets serialized is not trivial in CMake — and LLVM people aren't happy with me adding fake dependencies to enforce serialization. But even if we did that, there's still the general problem that other people will be unaware of the issue and will be hitting it in the future.
Therefore, I think it would be best to fix it on Sphinx side. The simplest solution would be to lock some file in the doctree directory. Considering the specific design, the environment pickle should be the best file to lock. However, that would require redesigning the application to keep the file open rather than closing it immediately after reading/writing (which would also be beneficial to avoiding race conditions).