Open mattstratton opened 5 years ago
mattstratton
commented
5 years ago FWIW, this is the change that seemed to make partials/head take much longer: https://github.com/devopsdays/devopsdays-theme/commit/f3e3487c7d035b71a90749bb4abe0e928ae6675b
All that was was REMOVING the og:updated_time from headers. That's SO WEIRD.
mattstratton
commented
5 years ago OK, reverting that change didn't affect the build time of partials/head, so it might just be my computer being slow. I'm going to reboot and will run templateMetrics again :)
mattstratton
commented
5 years ago Same results after a reboot on my side and a machine that isn't doing anything else, so I don't think the increase in time is related to that og:updated_time change, but maybe just a function of having one whole year's worth more files?
mattstratton
commented
5 years ago I suspect that this code (which exists on partials/head/open_graph.html and partials/head/twitter_cards.html is partially to blame:
{{- $e := (index $.Site.Data.events (index (split (.Permalink | relURL) "/") 2)) -}}On every single, welcome, talk, speaker, speakers, and program page, that query gets executed twice just to build the <head> meta tags.
One thing that could help (I don't quite understand how cumulative these things are) would be to combine all the partials in head/seo into a single partial; this way the single partial would be able to use one data query instead of having to do it multiple times for open graph AND for twitter code.
mattstratton
commented
5 years ago PR https://github.com/devopsdays/devopsdays-web/pull/7950 combines all the SEO templates into a single template; on my local machine this cut 30s off of the hugo build. If the PR passes, I'll merge it :)
mattstratton
commented
5 years ago PR #7950 has been merged :)
mattstratton
commented
5 years ago Just ran template metrics again, so these are the templates to continue to target:
cumulative average maximum
duration duration duration count template
---------- -------- -------- ----- --------
4m6.447973961s 1.825540547s 6.223745544s 135 program/single.html
2m28.885470424s 56.545943ms 268.589389ms 2633 speaker/single.html
1m31.734827266s 12.821079ms 4.069035612s 7155 partials/head.html
1m27.452857839s 12.222621ms 4.068451169s 7155 partials/head/seo.htmlFor the record, since these happen in parallel, it doesn't do a lot of good to optimize anything other than the program page first (it's the one that takes the longest). Even if we make speaker/single.html faster, for example, the entire build is still going to take a long time.
One thing that is weird is that the build above took 157798 ms, which is 2.6 minutes, but the program pages report as taking 4 minutes. Go figure.
mattstratton
commented
5 years ago Aha. The durations for template metrics are based on CPU time, not elapsed time.
mattstratton
commented
5 years ago I had a thought on how we could improve the performance of the speaker/single.html template, but it's not super awesome.
Right now, part of why it's so slow and intensive is this set of code:
<div class="speaker-bio-talks">
<h3>{{ .Title }} at {{ $e.city }} {{$e.year}}</h3>
<ul class="list-group">
{{- $.Scratch.Set "speaker" .File.BaseFileName -}}
{{ range where (where $.Site.Pages "Type" "talk") ".File.Dir" "="
(print "events/" $e.name "/program/") }}
<!-- Now we can display stuff! -->
{{- range .Params.speakers -}}
{{- if eq . ($.Scratch.Get "speaker") -}}
{{- $.Scratch.Set "display" "true" -}}
{{- end -}}
{{- end -}}
{{- if eq ($.Scratch.Get "display") "true" -}}
<a href = "{{ .Permalink }}" class= "list-group-item
list-group-item-action">{{ .Title }}</a>
{{ $.Scratch.Set "display" "false" }}
{{- end -}}
{{- end -}} <!-- end range where -->
</ul>
</div>
</div>Let's walk through this and understand what it's doing (fyi, this is what shows the "Michael Bluth at Ponyville 2020" widget with the talks at that event from this speaker):
$.Scratch.Set "speaker" .File.BaseFileNameThis takes the value of the .File.BaseFileName of the page being constructed (if the page is michael-bluth.md, the value is "michael-bluth") and shoves it into Scratch variable called "speaker".
range where (where $.Site.Pages "Type" "talk") ".File.Dir" "=" (print "events/"
$e.name "/program/")This executes a query against all the pages of type "talk" in the content/events/YYYY-city/program/ directory, which it will then iterate over.
range .Params.speakersThis query executes on every "talk" page it finds in the event's program directory. It steps through all the values in the array speakers that it finds in the frontmatter for that talk.
{{- if eq . ($.Scratch.Get "speaker") -}}
{{- $.Scratch.Set "display" "true" -}}
{{- end -}}Stepping through each of the values in the array, if the current value of the array item matches what is in Scratch for "speaker", it sets a value of "true" to the Scratch variable "display".
{{- if eq ($.Scratch.Get "display") "true" -}}
<a href = "{{ .Permalink }}" class= "list-group-item
list-group-item-action">{{ .Title }}</a>
{{ $.Scratch.Set "display" "false" }}
{{- end -}}If "display" is currently "true", it generates the HTML to create the link to the talk. It then sets "display" back to "false"
So, there are a few things that are less than efficient. One thing that comes to mind is that the Scratch variable of "display" may no longer be needed. We can possibly just do this:
{{- if eq . ($.Scratch.Get "speaker") -}}
<a href = "{{ .Permalink }}" class= "list-group-item
list-group-item-action">{{ .Title }}</a>
{{- end -}}I suspect there was a reason in the past to use the scratch variable for this (usually we use them because nesting if statements in Go templates is an issue for variable scope, but I don't see anything in here that would be an issue.
That being said, the biggest problem is that for every speaker, we have to query and load all the talks from that event. It's a small thing but it can definitely add up, as there are a lot of Speaker pages.
The possible solution is to add a bit of frontmatter to the speakername.md file, which would look like this:
talks = "[michael-bluth]"This would basically embed the talks that this speaker is giving at that event inside the frontmatter of the speaker page. So let's say that Michael Bluth is giving two talks: one by himself and one with George Bluth. The frontmatter would look like this:
talks = "michael-bluth", "george-and-michael-bluth"]In the template, we would no longer have to do the lookups at all on the pages of type talk; we would simply generate the HTML links while stepping through
.Params.talks.
But there's a problem. We need the title of the talk, and we won't have that from the frontmatter.
So basically, I've just realized this potential solution doesn't work. I'm keeping this in this comment in case it spurs another idea.
mattstratton
commented
5 years ago Spinning off older events to static had a non-trivial impact on the build, so yay.
Here's what it used to be:
cumulative average maximum
duration duration duration count template
---------- -------- -------- ----- --------
4m6.447973961s 1.825540547s 6.223745544s 135 program/single.html
2m28.885470424s 56.545943ms 268.589389ms 2633 speaker/single.html
1m31.734827266s 12.821079ms 4.069035612s 7155 partials/head.html
1m27.452857839s 12.222621ms 4.068451169s 7155 partials/head/seo.htmlnow it looks like this:
cumulative average maximum
duration duration duration count template
---------- -------- -------- ----- --------
1m20.521676812s 936.298567ms 2.475422152s 86 program/single.html
1m2.125752735s 38.254773ms 230.968171ms 1624 speaker/single.html
54.306623225s 12.24225ms 4.7990159s 4436 partials/head.html
51.763550073s 11.668969ms 4.797610796s 4436 partials/head/seo.html
19.808843368s 22.690542ms 4.829697529s 873 event/single.htmlI have no idea why the average duration for program/single.html is cut in half, but maybe the older program files were slower for some reason? LOL
mattstratton
commented
5 years ago I will say that thinking about this now and looking at the average duration being less than a second, archiving off to static is the most effective way of trimming the build; making the program page generate faster is less impactful than just having to do it less frequently?
tgross
commented
5 years ago I have no idea why the average duration for program/single.html is cut in half, but maybe the older program files were slower for some reason? LOL
That page iterates over all the talks and all the speakers, I think? So pulling a bunch of old content out of the build should really reduce the length of those loops.
mattstratton
commented
5 years ago Tim, you are a smart cookie. I had repressed the fact that I looped over ALL talks, not just the ones for that event :)
tgross
commented
5 years ago ~Hm, I know there's a subtle difference between .Site.Pages and .Pages: https://gohugo.io/variables/site/#site-pages-compared-to-pages But I don't think the program/single.html page is a "list page" as described there either so .Pages might not improve things (or even work?).~
After your edit this comment makes no sense. Serves me right for typing slowly :grinning:
mattstratton
commented
5 years ago NINJA EDIT!
mattstratton
commented
4 years ago Comparing the last time this was ran to today (which includes more archiving and a hugo upgrade that is supposed to improve performance)
Previous:
cumulative average maximum
duration duration duration count template
---------- -------- -------- ----- --------
1m20.521676812s 936.298567ms 2.475422152s 86 program/single.html
1m2.125752735s 38.254773ms 230.968171ms 1624 speaker/single.html
54.306623225s 12.24225ms 4.7990159s 4436 partials/head.html
51.763550073s 11.668969ms 4.797610796s 4436 partials/head/seo.html
19.808843368s 22.690542ms 4.829697529s 873 event/single.htmlCurrent:
cumulative average maximum
duration duration duration count template
---------- -------- -------- ----- --------
1m10.52828606s 892.763114ms 2.573561246s 79 program/single.html
51.624310039s 37.166529ms 167.809596ms 1389 speaker/single.html
15.819940587s 3.977857ms 2.813484716s 3977 partials/head.html
13.061653031s 3.284297ms 2.812543642s 3977 partials/head/seo.html
12.625062601s 14.012278ms 2.95837693s 901 event/single.html
10.194041982s 2.648491ms 69.08567ms 3849 partials/sponsors.html
8.29566866s 6.232658ms 133.027297ms 1331 talk/single.html
5.043203144s 60.038132ms 153.749825ms 84 speakers/single.htmlDefinitely seems like the latest hugo upgrade improved performance? I am also wondering if things got better since the seo.html no longer does filesystem checks.
mattstratton
commented
4 years ago In the template, we would no longer have to do the lookups at all on the pages of type talk; we would simply generate the HTML links while stepping through .Params.talks.
But there's a problem. We need the title of the talk, and we won't have that from the frontmatter.
This might actually work - if we use .GetPage on with the title of the talk from the speaker's frontmatter, we can retrieve the talk title, without having to load all the talks, etc.
Continuing the work @tgross did in https://github.com/devopsdays/devopsdays-theme/issues/643
So yeah, that program page, wut.
Although the
partials/headstuff has gotten much worse somehow. This is what it looked like a year ago:So building the
headfor the pages used to take a total of 16 seconds, and now it takes over 2 minutes. Hmm.