Open jkrumbiegel opened 3 weeks ago
I like the idea!
This seems pretty great! I need to find some time to go through the implementation, but generally I don't think I have any reservations about it (except the global state). This is looking pretty good already!
Yeah maybe scoped values would be appropriate here? Although they might be too new for Documenter, that's why I didn't try them.
Another question I had was if the implementation could be pushed outside of all the runner functions. But I needed some info that was already available inside, like line numbers, and I didn't want to duplicate that. Maybe there's a trick how the timing code can be placed more elegantly.
I've run this branch on my own documentation. Everything seems to build correctly and the timing output is also very informative. I will mention however that the docs I ran don't have any doctesting so I can't speak if that works or not with this branch.
My only thought was that since the timing table output can get quite large, maybe there should be an option to disable the benchmarking or to collapse the table output to be by page only.
I would probably make it opt-in with an option in makedocs
A simple api that makes it less global could be to pass in a TimerOutput
or simply a Ref()
via a keyword which at the end holds all the measurements. The user could then do whatever with it, just print it as is, or postprocess it. And you don't have to change the return type of makedocs.
I thought some useful post processing might be to remove all code blocks under some threshold of either absolute time or percentage of page. Because you're going to care about the 20% of code using 80% of the time.
The way I have it refactored now is that there are two levels (aside from no timings) and with full timings you get timings for every block, for the other one just for every page.
makedocs(;
sitename = "Some Site",
timings = Documenter.BasicTimings,
)
────────────────────────────────────────────────────────────────────────
Section ncalls time %tot
────────────────────────────────────────────────────────────────────────
ExpandTemplates: expanding markdown templates. 1 3.22s 98.3%
index.md 1 3.22s 98.3%
RenderDocument: rendering document. 1 54.5ms 1.7%
SetupBuildDirectory 1 2.21ms 0.1%
Doctest: running doctests. 1 40.6μs 0.0%
CheckDocument: running document checks. 1 32.8μs 0.0%
CrossReferences: building cross-references. 1 24.6μs 0.0%
────────────────────────────────────────────────────────────────────────
makedocs(;
sitename = "Some Site",
timings = Documenter.FullTimings,
)
────────────────────────────────────────────────────────────────────────
Section ncalls time %tot
────────────────────────────────────────────────────────────────────────
ExpandTemplates: expanding markdown templates. 1 3.22s 98.6%
index.md 1 3.22s 98.6%
L1-4 @example sleep(2) sleep(0.1) 1 2.11s 64.5%
L6-12 @example sleep(1) # more # code # m... 1 1.00s 30.7%
L14-16 @setup name 1 + 1 1 835μs 0.0%
RenderDocument: rendering document. 1 44.8ms 1.4%
SetupBuildDirectory 1 2.53ms 0.1%
Doctest: running doctests. 1 26.0μs 0.0%
CheckDocument: running document checks. 1 9.08μs 0.0%
CrossReferences: building cross-references. 1 2.79μs 0.0%
────────────────────────────────────────────────────────────────────────
I was already halfway started to make a more complicated system where you can filter even more and come up with custom selections, but then I thought that's probably way overkill.
Makie has switched back to Documenter(Vitepress), so I was reminded of the issue that I have no insight into the build time. We have tons of examples that can also be rather heavy, so my idea was to list all of them sorted by time using TimerOutputs.jl. This PR is for now just a draft of how this could look.
Here's an example output from a reduced run of the Makie docs, where one can see that the intervalslider page needs half the time due to a long video being rendered. (The lines can usually be shown but I'm mostly using a modified example block where the matching of the code back to the file breaks)