Slow page load times for /whatsnew/ pages in GA

alexgibson commented 5 years ago

I can't reproduce any real issues when testing manually, but I noticed in GA that /whatsnew/ is showing some really high page load times, with averages often above the 10 second mark.

I'm not clear on what's going on here, and can't currently reproduce the issue. Page Speed Insights also doesn't reveal any obvious problems. It could be that there's an error with one of our geo-location calls, or an issue with some piece of conditional logic slowing the page down. I'm not really sure right now. The /whatsnew/ page gets such a high proportion of traffic compared to other pages, that it's currently lowering our average page load score across the site, so is worth paying attention to.

The upcoming /whatsnew/ page for FF 63 is much simpler in what it does, so it may be worth keeping an eye on how it plays out in the next release. If we still see slow loading times, it may point to a different issue.

Pageviews	Pageviews	Avg. Page Load Time (sec)
/en-US/firefox/62.0/whatsnew/?oldversion=61.0.2	23,739,290	14.48	4 total	91
/de/firefox/62.0/whatsnew/?oldversion=61.0.2	8,600,991	13.21	5 total	84
/fr/firefox/62.0/whatsnew/?oldversion=61.0.2	4,887,390	16.25	6 total	83
/es-ES/firefox/62.0/whatsnew/?oldversion=61.0.2	3,670,728	18.66	5 total	84
/pt-BR/firefox/62.0/whatsnew/?oldversion=61.0.2	2,846,644	19.81

stephaniehobson commented 5 years ago

No luck reproducing this with webpagetest from an IP in Brazil with en-US or pt-BR. I wonder if the missing link is something that product is doing.

stephaniehobson commented 5 years ago

I got a load time of (estimated) 14 seconds when updating from 61 to 62 in a Windows 10 VM and 11.44 reloading with an empty cache (measured with network tool).

GA doesn't seem to report Windows users have a worse experience over all though.

stephaniehobson commented 5 years ago

Also, it looks like there's only a 1.5 second (average) increase between 61's simple version and 62's more complex. This page was pretty slow before 62 :(

15.10s became 16.65s compared to the site average of 10.69s for Firefox.

Still wondering if there's a product reason. Maybe it needs to update safe browsing lists or something when a new version is opened for the first time?

stephaniehobson commented 5 years ago

I've spent enough time down this particular rabbit hole for now.

alexgibson commented 5 years ago

Myself and @rraue spent time time looking at this in GA today to try and answer some questions:

Is /whatsnew nearly always slow to load, in comparison to other pages on the site?

Yes. Average page load time on mozorg (excluding /whatsnew) is around 8 seconds. The average load time for /whatsnew itself is around 14 seconds, so considerably slower.

Does /whatsnew for 63.0 load any faster than /whatsnew for 62.0?

Yes. The /whatsnew page for Firefox 63.0 saw a 23% improvement in page load times, from ~17 to ~13 seconds after it was simplified. 13 seconds is still considerably higher than the site average however.

How long has /whatsnew been slow to load for, is there a date where this noticeably begins?

We looked back over the last 3 years, and saw some large spikes related to different Firefox releases and /whatsnew designs. The /whatsnew page for 62.0 seemed to indicate some particularly high page load times. Page design definitely seems to be a factor, but not the root of the issue as we'll see below.

Is there much non-direct traffic to /whatsnew? How do load times compare to direct?

There's not much non-direct traffic to the page understandably, so instead we did a comparison of FF and non-FF traffic. FF users see an average of 14 second load times, and non-FF users see an average of around 3 seconds. This upholds the theory that external factors from the product update process may be contributing to higher page load times.

What are the slowest loading countries for /whatsnew?

Many countries see far worse page load times than the average. Here's a list of the poorest performers, with page load times ranging between 28 seconds to well over 1 minute :/

Pageviews	Country	Pageviews	Avg. Page Load Time (sec)
/es-ES/firefox/62.0/whatsnew/?oldversion=61.0.2	Venezuela	371,551	112.96%
/es-CL/firefox/62.0/whatsnew/?oldversion=61.0.2	Chile	167,835	203.55%
/fr/firefox/62.0/whatsnew/?oldversion=61.0.2	Côte d’Ivoire	52,207	150.12%
/fr/firefox/62.0/whatsnew/?oldversion=61.0.2	Cameroon	38,692	223.22%
/en-US/firefox/62.0/whatsnew/?oldversion=61.0.2	Estonia	28,585	105.25%
/en-US/firefox/62.0/whatsnew/?oldversion=61.0.2	Rwanda	19,313	102.23%
/en-US/firefox/62.0/whatsnew/?oldversion=61.0.2	Zambia	17,383	103.45%
/fr/firefox/62.0/whatsnew/?oldversion=61.0.2	Mali	13,803	106.91%
/fr/firefox/62.0/whatsnew/?oldversion=61.0.2	Benin	13,512	192.90%
/fr/firefox/62.0/whatsnew/?oldversion=61.0.2	Togo	13,412	219.35%
/es-ES/firefox/62.0/whatsnew/?oldversion=61.0.2	Cuba	12,935	175.67%
/es-AR/firefox/62.0/whatsnew/?oldversion=61.0.2	Ecuador	10,372	104.08%
/es-AR/firefox/62.0/whatsnew/?oldversion=61.0.2	Venezuela	9,627	110.09%
/en-US/firefox/62.0/whatsnew/?oldversion=61.0.2	Cameroon	9,302	104.31%
/en-US/firefox/62.0/whatsnew/?oldversion=61.0.2	Brunei	9,285	263.98%
/en-US/firefox/62.0/whatsnew/?oldversion=61.0.2	Belarus	8,662	383.87%
/es-MX/firefox/62.0/whatsnew/?oldversion=61.0.2	El Salvador	7,077	107.09%
/zh-CN/firefox/62.0/whatsnew/?oldversion=61.0.2	Australia	6,832	363.14%
/zh-CN/firefox/62.0/whatsnew/?oldversion=61.0.2	Singapore	6,471	608.53%
/en-US/firefox/62.0/whatsnew/?oldversion=61.0.2	Uruguay	6,176	378.89%

Are conversions for slower loading countries noticeably worse?

This was hard to dig into during the meeting as many /whatsnew pages have different states that can vary on both country and locale. For some countries, we did see worse click through rates (e.g. /pt-BR/ FxA form button conversions we're 0.99% compared to an average of 2.28%). There was a lot of noise here though for some other countries and data that didn't make as much sense, so we couldn't be conclusive. I think it's probably safe to say however, that 30s page load times will inevitably perform worse than fast loading pages.

Conclusions

We should be thinking really hard about designing our /whatsnew pages to be really fast and performant, even more so than our regular web pages. The slow loading times we see above are large enough to skew the average for our entire website considerably. I think much of this may be related to external factors caused by the product update process (browser restarting, updating, multiple tabs reloading etc), but for certain countries this can have a really crippling effect on page load time, and I think it's safe to say this will naturally hurt conversions. These are all things we should consider when updating /whatsnew for future releases, and could also form the basis of some interesting experiments to improve page load times in poorer areas of the world.

/cc @djst @dzingeek

stephaniehobson commented 5 years ago

Thoughts on things we could do for performance on this page specifically (not all of these would be quickly/easily done on the entire site).

Don't load the site.js common bundle (we don't even have the nav on this page, only need JS for language and analytics)
Can we subset Protocol?
Can we inline the page specific CSS?
Is this page indexed? We could save some HTML bytes but not including the <link rel="alternate".

We could also file a performance bug with product.

stephaniehobson commented 5 years ago

Thinking we might have more users with bad bandwidth hitting What's New than we do moz.org in general I did a speed comparison for users in LA.

I picked LA because I figured Google is more likely to be accurate with the geolocation in the USA and that the city has a high enough urban population that any rural outliers included in it would have a smaller effect.

What's New is still slower than the site average.

stephaniehobson commented 5 years ago

Depends on https://github.com/mozilla/bedrock/issues/6504

mozilla / bedrock

Slow page load times for /whatsnew/ pages in GA #6296

Conclusions