Open JulienPalard opened 9 months ago
Building only the changed branches and languages, how does it sound?
I don't have much information about the server but here are some ideas:
git pull
everytime we build; we can instead git fetch
in each repo and see if there are any changes, and if that's the case we can pull and build.Additional question for @JulienPalard: Are non-bugfix and non-stable releases only rebuilt manually or is there a cron?
Are non-bugfix and non-stable releases only rebuilt manually or is there a cron?
EOL
and security-fixes
branches are only built manually, yes.
What do you think about the idea in general @JulienPalard?
I don't have much information about the server
(see the git_clone function in the script).
Building only the changed branches and languages, how does it sound?
Maybe, yes. But to do it cleanly we should only rebuild if the Doc/
directory changed, but doable and cheap as we keep the cpython clone.
Currently a git log shows that (for the Doc/ directory) changes are:
A change cpython side should trigger a rebuild of all languages for the given branch (while a change for a language should not trigger a rebuild for all branches: translation repo has a translation branch per cpython branch).
Translation side I expect less changes, except for a few crons like for python-docs-ja which synchronizes daily. Also changes are for a single branch so they invalidate a single cpython branch.
So instead of rebuilding 78 docs per day, just looking at the cpython side, we'd rebuild like 26 docs daily (main and 3.11) which should take like 9h.
About how to do this, we could store at build time the cpython commit sha and the translation commit sha in a file, and at build time call a dedicated function to check the shas against the repositories to see if a build is needed or not.
Seems doable.
@JulienPalard Someone opened an issue on Sphinx in order to know whether we could speed-up things on our side. I am not quite confident in that since most of the build time is actually the PDF build. It is possible to know why make all-pdf
is so slow? is it only because there are a lot of pages to write? or is it because of the underlying language? (iirc, jp builds don't build with plain pdflatex).
Another possibility is to change the way the PDF are created. We (Sphinx) could technically add a builder which, instead of outputting LaTeX code, outputs some other kind of typesetting language that can still be converted to PDF faster than LaTeX.
Hi @picnixz, thanks for jumping in!
It is possible to know why
make all-pdf
is so slow?
I'm just a user of LaTeX, I don't know much. I know our biggest file is library.tex
with 288k lines of LaTeX, generating a 11MB PDF file.
make all-pdf
runs latexmk
for each .tex
files, which in turn runs xelatex
. So I tried just running time latexmk -pdf -dvi- -ps- library.tex
, it takes 1 min 16s (on the same laptop used for the timings in the first post). same timing for the underlying command: xelatex -recorder library.tex
.
Those timings can be reproduced on a cpython clone after building latex file:
make -C Doc/build/latex/ clean
time make -C Doc/build/latex/ library.pdf # 4 min 30s
time (cd Doc/build/latex; make clean; XINDYOPTS="-L english -C utf8 -M sphinx.xdy" latexmk -pdf -dvi- -ps- library.tex) # 4 min 13s
time (cd Doc/build/latex; xelatex -recorder library.tex` # 1 min 16s
It reminds me of something about running it in a loop until the output does not changes..., which strace confirms:
$ grep bin/xelatex make.strace
15622 1696969290.322763 execve("/usr/bin/xelatex", ["xelatex", "-recorder", "library.tex"], 0x55b2cfd2a5d8 /* 90 vars */) = 0
15719 1696969388.530816 execve("/usr/bin/xelatex", ["xelatex", "-recorder", "library.tex"], 0x5651c759b5d8 /* 90 vars */) = 0
15766 1696969471.969531 execve("/usr/bin/xelatex", ["xelatex", "-recorder", "library.tex"], 0x55ddc43a15d8 /* 90 vars */) = 0
Which also can be confirmed by reading the console log (I choose a smaller file to get readable output):
------------
Run number 1 of rule 'pdflatex'
------------
------------
Running 'xelatex -recorder "howto-cporting.tex"'
------------
[...]
Rule 'pdflatex': Reasons for rerun
Changed files or newly in use/created:
howto-cporting.aux
howto-cporting.ind
howto-cporting.out
howto-cporting.toc
------------
Run number 2 of rule 'pdflatex'
------------
------------
Running 'xelatex -recorder "howto-cporting.tex"'
------------
Also noticed, from xelatex
output:
LaTeX Warning: Label(s) may have changed. Rerun to get cross-references right.
Package rerunfilecheck Warning: File `library.out' has changed.
(rerunfilecheck) Rerun to get outlines right
(rerunfilecheck) or use package `bookmark'.
this is probably less than ideal.
I tried to use perf
to get some insights about what's slow inside xetex, if I read this correctly that's the parsing of the input file that's slow, can be reproduced using:
sudo sysctl kernel.perf_event_paranoid=-1
sudo apt install texlive-binaries-dbgsym # needs `deb https://deb.debian.org/debian-debug trixie-debug main`
time (cd Doc/build/latex/; make clean; perf record -F 1000 -g --call-graph dwarf -o perf.data xelatex -recorder library.tex)
(cd Doc/build/latex/; perf report --tui)
it looks like this:
- 80.59% 0.00% xelatex xetex [.] main ▒
- main ▒
- 80.54% mainbody ▒
- 80.37% maincontrol ▒
- 35.42% getxtoken ▒
- 18.35% expand ▒
- 12.60% conditional ▒
+ 9.10% scanint ▒
+ 1.37% passtext ▒
+ 3.13% expand ▒
0.54% passtext ▒
- 14.03% macrocall ▒
6.20% getnext ▒
1.92% endtokenlist ▒
+ 2.67% getnext ▒
- 35.25% prefixedcommand ▒
- 29.63% zscantoks ▒
- 26.62% expand ▒
- 19.90% macrocall ▒
- 10.50% getnext ▒
3.33% endtokenlist ▒
1.77% endtokenlist ▒
0.63% getavail ▒
+ 5.68% conditional ▒
1.87% getnext ▒
+ 3.12% zdoregistercommand ▒
0.52% zeqdefine ▒
+ 3.87% measure_native_node ▒
0.86% unsave ▒
+ 0.60% zshipout ▒
+ 0.58% endgraf.part.0 ▒
0.51% zbeginbox ▒
is it only because there are a lot of pages to write?
Probably yes, library.pdf
is 2318 pages, or lots of latex to parse if my perf reading is good.
or is it because of the underlying language? (iirc, jp builds don't build with plain pdflatex).
Looking at the server logs, yes jp
is slower than all the others. It takes like 1h30 while the other take like 30mn (for a full build of text, html, pdfs, ...). All builds are using xelatex while jp builds are using lualatex
. As it's the only one I won't focus on this one.
Another possibility is to change the way the PDF are created. We (Sphinx) could technically add a builder which, instead of outputting LaTeX code, outputs some other kind of typesetting language that can still be converted to PDF faster than LaTeX.
It would be possible to build PDF using weasyprint.
It looks like a potential alternative for PDFs building is using Rinohtype. It provides a drop-in replacement for Sphinx PDF builder and is a direct PDF builder. Still in beta though.
Thank you very much for your report !
Now I'm a bit confused about these timings:
time make -C Doc/build/latex/ library.pdf # 4 min 30s time (cd Doc/build/latex; make clean; XINDYOPTS="-L english -C utf8 -M sphinx.xdy" latexmk -pdf -dvi- -ps- library.tex) # 4 min 13s time (cd Doc/build/html; xelatex -recorder library.tex` # 1 min 16s
Maybe I misunderstood, but why does the second command takes 4 min 13s but you told me it took 1min 16s? I also think the last command should be cd Doc/build/latex
instead of cd Doc/build/html
.
if I read this correctly that's the parsing of the input file that's slow
Yes, it's because of how LaTeX is structured with this token based approach + expansions. Depending on how expansions are done, it may enormously slow down the build. I am wondering whether breaking down library.tex
into smaller pieces that are put together at the end would improve or not the timings.
In conclusion, I don't think we could do much except:
By the way, how much is allocated to build the documentation on Python servers? because a straightforward solution is simply to allocate more resources and run everything in parallel (but again, funds are required and I don't know if the server is like a super-mega-huge-powerful server). In the end, if you need to rebuild 26 docs per day (assuming you can reduce from 70+), the you "simply" need a more powerful unit.
Now I'm a bit confused about these timings:
time make -C Doc/build/latex/ library.pdf # 4 min 30s time (cd Doc/build/latex; make clean; XINDYOPTS="-L english -C utf8 -M sphinx.xdy" latexmk -pdf -dvi- -ps- library.tex) # 4 min 13s time (cd Doc/build/html; xelatex -recorder library.tex` # 1 min 16s
Maybe I misunderstood, but why does the second command takes 4 min 13s but you told me it took 1min 16s?
That's because latexmk
runs xelatex
in a while True:
loop until the output stabilizes. Looks like it had to run it 4 times to reach stabilization (I have a few paragraph in my last message about it, search for 'strace' and 'rerun').
I really don't know if there's a way to forge a latex file that necessitates less re-runs.
I also think the last command should be
cd Doc/build/latex
instead ofcd Doc/build/html
.
Probably just me manually fixing the commands for readability (and breaking them while doing so, haha).
That's because latexmk runs xelatex in a while True: loop until the output stabilizes
Ah sorry! yes I overlooked that (I was a bit confused actually because I assumed that the ~1min was the output of time
). Now, the question is: is it possible to avoid using xelatex
actually and only pdflatex
?
Also, since it tells us "or use package bookmark
", maybe this could solve the issue (though I don't know how). Nevertheless, yet another alternative is to run the latex command only once and check whether more compilation is needed (and not let latexmk decides by itself). I'm not sure whether the 4 runs are actually needed to solve all the references (in general, we need 2 reruns but here I'm wondering why we actually need 4).
Btw, I'm sorry but I cannot really reproduce it myself because I need to install fonts that I don't have (and move them around files + adding paths or so) (but since you've got everything running on your side + timings are for your machine, the comparison is more fair).
IIRC we use xetex instead of pdflatex for its unicode awareness.
Tried:
$ make -C build_root/cpython/Doc PYTHON=build_root/venv-3.11/bin/python SPHINXBUILD=build_root/venv-3.11/bin/sphinx-build BLURB=build_root/venv-3.11/bin/blurb VENVDIR=build_root/venv-3.11 'SPHINXOPTS=-D latex_engine=pdflatex -q' SPHINXERRORHANDLING= autobuild-stable
[...]
LaTeX Warning: Hyper reference `howto/regex:the-backslash-plague' on page 6 und
efined on input line 816.
[6]
! LaTeX Error: Unicode character ſ (U+017F)
not set up for use with LaTeX.
See the LaTeX manual or LaTeX Companion for explanation.
Type H <return> for immediate help.
...
l.962 Latin small letter dotless i), ‘ſ
’ (U+017F, Latin small letter lo...
?
Since https://github.com/python/docsbuild-scripts/pull/171 has been merged, there has been a build taking 13.6 hours. That's better, probably still room for enhancements.
I changed the build cron so it starts hourly, so instead of doing nothing for 24h-13.6h the script checks if there's something to build.
In the logs here's what I see:
2023-10-27 04:56:30,401 INFO en/3.12: Nothing changed, no rebuild needed.
2023-10-27 04:56:31,868 INFO zh-tw/3.13: Should rebuild: new translations (from a4719a1e1605163e886fad5c3783bdac250f0db9 to f24fd11929f41176367f42559c62b808e7468b2d)
2023-10-27 06:15:32,386 INFO zh-cn/3.13: Should rebuild: new translations (from 99da58558f6c81e2808bbe85307f5d480c9ec096 to ef8a1e2bd3a47f9a5d21c7f6495d3f525f3c66e6)
2023-10-27 07:30:51,890 INFO uk/3.13: Nothing changed, no rebuild needed.
2023-10-27 07:30:52,196 INFO tr/3.13: Nothing changed, no rebuild needed.
2023-10-27 07:30:52,494 INFO pt-br/3.13: Nothing changed, no rebuild needed.
2023-10-27 07:30:52,804 INFO pl/3.13: Nothing changed, no rebuild needed.
2023-10-27 07:30:53,072 INFO ko/3.13: Nothing changed, no rebuild needed.
2023-10-27 07:30:53,989 INFO ja/3.13: Should rebuild: new translations (from 32e85a08b356b6faa3c5784f1d5bdea15ec2e4f4 to 260a16dd7cb834bcffe2bee64375f6e8d43f1b35)
2023-10-27 08:51:27,928 INFO it/3.13: Nothing changed, no rebuild needed.
2023-10-27 08:51:28,353 INFO id/3.13: Nothing changed, no rebuild needed.
2023-10-27 08:51:28,685 INFO fr/3.13: Nothing changed, no rebuild needed.
2023-10-27 08:51:29,262 INFO es/3.13: Nothing changed, no rebuild needed.
2023-10-27 08:51:29,363 INFO en/3.13: Nothing changed, no rebuild needed.
2023-10-27 09:07:07,306 INFO zh-tw/3.11: Nothing changed, no rebuild needed.
2023-10-27 09:07:08,020 INFO zh-cn/3.11: Nothing changed, no rebuild needed.
2023-10-27 09:07:08,501 INFO uk/3.11: Nothing changed, no rebuild needed.
2023-10-27 09:07:08,992 INFO tr/3.11: Nothing changed, no rebuild needed.
2023-10-27 09:07:09,540 INFO pt-br/3.11: Nothing changed, no rebuild needed.
2023-10-27 09:07:09,947 INFO pl/3.11: Nothing changed, no rebuild needed.
2023-10-27 09:07:10,255 INFO ko/3.11: Nothing changed, no rebuild needed.
2023-10-27 09:07:11,250 INFO ja/3.11: Nothing changed, no rebuild needed.
2023-10-27 09:07:11,512 INFO it/3.11: Nothing changed, no rebuild needed.
2023-10-27 09:07:11,767 INFO id/3.11: Nothing changed, no rebuild needed.
2023-10-27 09:07:12,113 INFO fr/3.11: Nothing changed, no rebuild needed.
2023-10-27 09:07:12,820 INFO es/3.11: Nothing changed, no rebuild needed.
2023-10-27 09:07:12,908 INFO en/3.11: Nothing changed, no rebuild needed.
2023-10-27 09:07:15,611 INFO zh-tw/3.12: Should rebuild: new translations (from a4719a1e1605163e886fad5c3783bdac250f0db9 to eaccd60ee755daee77d4fe24707c1a2ec6059bfb)
2023-10-27 09:49:24,057 INFO zh-cn/3.12: Nothing changed, no rebuild needed.
2023-10-27 09:49:24,383 INFO uk/3.12: Nothing changed, no rebuild needed.
2023-10-27 09:49:24,759 INFO tr/3.12: Nothing changed, no rebuild needed.
2023-10-27 09:49:25,169 INFO pt-br/3.12: Nothing changed, no rebuild needed.
2023-10-27 09:49:25,541 INFO pl/3.12: Nothing changed, no rebuild needed.
2023-10-27 09:49:25,820 INFO ko/3.12: Nothing changed, no rebuild needed.
2023-10-27 09:49:26,652 INFO ja/3.12: Nothing changed, no rebuild needed.
2023-10-27 09:49:26,920 INFO it/3.12: Nothing changed, no rebuild needed.
2023-10-27 09:49:27,175 INFO id/3.12: Nothing changed, no rebuild needed.
2023-10-27 09:49:27,500 INFO fr/3.12: Nothing changed, no rebuild needed.
2023-10-27 09:49:28,223 INFO es/3.12: Nothing changed, no rebuild needed.
2023-10-27 09:49:28,313 INFO en/3.12: Nothing changed, no rebuild needed.
2023-10-27 09:49:31,085 INFO zh-tw/3.13: Should rebuild: new translations (from f24fd11929f41176367f42559c62b808e7468b2d to eaccd60ee755daee77d4fe24707c1a2ec6059bfb)
2023-10-27 11:08:41,249 INFO zh-cn/3.13: Should rebuild: Doc/ has changed (from a254120f2f1dd99fa64f12594d1ed19c67df7d64 to 74f0772892c85b6e7bdfa0f44a5ff89002b0734d)
2023-10-27 12:31:52,133 INFO uk/3.13: Should rebuild: Doc/ has changed (from 3f84a19e6291db682fc9a570e7612e80e2ffbbb5 to 74f0772892c85b6e7bdfa0f44a5ff89002b0734d)
2023-10-27 12:35:32,533 INFO tr/3.13: Should rebuild: Doc/ has changed (from 3f84a19e6291db682fc9a570e7612e80e2ffbbb5 to 74f0772892c85b6e7bdfa0f44a5ff89002b0734d)
2023-10-27 13:08:20,924 INFO pt-br/3.13: Should rebuild: Doc/ has changed (from 3f84a19e6291db682fc9a570e7612e80e2ffbbb5 to 74f0772892c85b6e7bdfa0f44a5ff89002b0734d)
The last https://docs.python.org/3/ build was at Dec 07, 2023 (12:24 UTC).
The 3.12.1 release was at something like Dec 08, 2023 (00:45 UTC).
As there's no public logs yet (https://github.com/python/docsbuild-scripts/issues/174), I'm curious why there's been no build yet from the hourly cron. Is the queue full, or maybe something else up?
This relates to https://discuss.python.org/t/python-3-12-1-now-available/40603/2?u=hugovk: a request for release announcements to include the changelog, but the changelog at https://docs.python.org/3/whatsnew/changelog.html still shows "Next" and not "3.12.1".
@hugovk @JulienPalard What are the next steps needed for this?
I see the last build on Last updated on Dec 11, 2023 (04:33 UTC)
, so the build is running, still slow though.
It now says Last updated on Dec 12, 2023 (08:00 UTC)
on https://docs.python.org/3/
If it took 27.5 hours, what language/versions has it been building to delay it?
There have been changes to 3.11-3.13 in the past day, but a week or more for 3.10 and older.
Since we stopped rebuilding when it's not needed, we're maintaining a file with some infos:
it's not that readable but may help a bit.
I can paste some logs here:
so yes, there's still almost 24h between any builds.
In 14 days we've seen only 45 build being skipped vs 449 rebuilds being needed:
mdk@docs:~$ zgrep 'no rebuild' /var/log/docsbuild/docsbuild.log.*.gz | wc -l
45
mdk@docs:~$ zgrep 'Should rebuild' /var/log/docsbuild/docsbuild.log.*.gz | wc -l
449
is is partly due to some translation repos having a cron commiting every day (like transifex pulls) even if there's no new translations in the files.
But the root cause of slow build time is probably more https://github.com/python/docsbuild-scripts/issues/169#issuecomment-1759197196 (latexmk running xelatex in a loop until the result stop changing).
@ewdurbin what "CPU" do we have on the machine? I don't see usefull info on /proc/cpuinfo
.
But the root cause of slow build time is probably more https://github.com/python/docsbuild-scripts/issues/169#issuecomment-1759197196 (latexmk running xelatex in a loop until the result stop changing).
How important are the latex builds?
We may want to look at additional ways of optimizing the latex builds. https://blog.martisak.se/2023/10/01/compiling/
How important are the latex builds?
PDF builds are based on LaTeX builds.
When PDFs are unavailable we have user complaints (typically on docs@).
Yeah, we definitely need PDFs.
I wonder if it makes sense to try building PDF with an alternate tool, ~such as pyppeteer which is used by Jupyter book. https://jupyterbook.org/en/stable/advanced/pdf.html#build-a-pdf-from-your-book-html~ playwright-python.
Shotscraper could be another approach: https://shot-scraper.datasette.io/en/stable/pdf.html
PDF shoud provide table of contents and index. So ePub is better than html for converting into PDF.
But I don't know good tool to convert ePub to PDF. Maybe, Calibre?
mdk@docs:~$ apt-cache policy latexmk
latexmk:
Installed: 1:4.67-0.1
Candidate: 1:4.67-0.1
Version table:
*** 1:4.67-0.1 500
500 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 Packages
100 /var/lib/dpkg/status
mdk@docs:~$ apt-cache policy texlive-xetex
texlive-xetex:
Installed: 2019.20200218-1
Candidate: 2019.20200218-1
Version table:
*** 2019.20200218-1 500
500 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 Packages
100 /var/lib/dpkg/status
(One comment: IIRC, at least one reason we do full rebuilds of the docs and not just incremental builds was that there were problems when a documentation page was deleted by a change and then would not automatically get deleted on-line. I'm not sure if that still is an issue.)
Today we're building 6 versions for 13 languages (that's 78 builds).
I'm doing some tests on my machine to get an idea:
make html
composed of:make text
make latex PAPER=a4
make all-pdf
(inbuild/latex/
) (less if run again, like 1 s if not removing PDFs, or 1 min after removing PDFs). Can be cut down to 6 min with-j 4
.So a complete rebuild should take ~27h (on my machine, the server may have a different CPU).