Expanding Client Timings to include Core Web Vitals

[Turnerj]

Was reading an article about capturing Core Web Vitals (eg. Cumulative Layout Shift, Largest Contentful Paint, First Input Delay) and it might be a good fit with the other client timings captured. The article goes into the JS changes required to capture the information and then it would just be a matter of storing and displaying it.

Is this a type of change that you feel may be a good fit for the library? I'd be happy to implement if you think it is a good idea.

Here are the Pros/Cons I see:

Pros

• The Core Web Vitals are statistics that are becoming an increasing focus for Google and will be a ranking signal from May 2021
• The data itself does give useful insights to problems with the content rendered to the page and possibly other bottlenecks that existing client timings don't show

Cons

• More code / higher maintenance burden
• Core Web Vitals may be important now but may itself be a bit of a whack-a-mole with Google potentially adding new/different ones or changing how it measures them

I imagine for SO it may be useful as you've spent a bunch of time making things fast, you wouldn't want some quirk of a page to make it still feel slow to a user. 🙂

[NickCraver]

I'd love to get this data, but there is one more collection problem to solve: ironically or not...timing. The current timings are safely captures at the point we need because they by-necessity happen before the point they are captured (e.g. on load).

These other (currently collected) events, especially if they're bad, can happen much later. We'd have to delay collection until the last event was ensured further down the pipe and that would mean either splitting render and collection (1 AJAX request today), or delay MiniProfiler showing up until that point to maintain 1 request. IMO, that delay is not appealing, but then again doubling request isn't either.

The delay option I don't think is awesome - MiniProfiler itself appearing slow (from a user experience standpoint) or even missing client timings entirely because a user navigated away before the send (which would now be in a larger window) are decent downsides. Depending on how bad the page is with ads loading, etc. and that experience (especially on mobile), the impact could be a large one.

A theoretical second request to "update" timings that exist has the main downside of "it's another request", and that's per-page we load or are sampled on. For most people it's lightweight but it's still taking 1 page into 3 requests today (1 for the page itself, 1 for MiniProfiler/client timings, and 1 for these additional timings). I'm thinking of this at Stack Overflow scale and have bias against it when maybe it's not that big of an issue for the wider audience...especially if it was something you opted into.

The other problem set is the storage providers here: when storing timings we have fields/tables/documents/indexes, etc. so we'd either have to a) not store this, or b) figure out a migration strategy. While I did clean up the storage a lot in v3 -> v4, the net results didn't change and required no existing-user migration. So we'd have to think about that thus-far-punted problem.

Thoughts on solutions/approaches to any of the above? I'm very much just brain dumping the to-solves here!

[Turnerj]

Yeah, you're 100% right - a delay or a second request both aren't great solutions. Going through the different stats:

• Largest Contentful Paint (LCP): The Web Vitals implementation stops after the first user input (where input is keydown or click - not scroll - they have an issue that briefly covers why).
• Cumulative Layout Shift (CLS): The Web Vitals implementation doesn't have a stop condition - layout shifts could happen at any point of time in the page. Realistically though, unless the website is made poorly, this nearly could be assumed at or shortly after the Window.Load event.
• First Input Delay (FID): The Web Vitals polyfill implementation seems to stop at any user input at all. Of the 3 stats, it is probably the most likely to happen first.

With this in mind, delaying wouldn't work at all as the delay could in theory be indefinite. Next to storing the result to send on the next page request (complicated and messy), we would need to do a second request unless we fundamentally change how MiniProfiler passes around data.

With a second request, like you've mentioned, it needs to be opt-in at minimum. It may even be useful building in some level of sampling for the data too. Some of the pressure from a second request would be relieved in a world where HTTP2 was the norm (less TCP connections, header compression etc) but it still means more work for the server to do than with 1 request for MiniProfiler.

I'm not going to go a huge amount into detail with the other approach as it is a fairly big change with many other issues but... If MiniProfiler didn't do a request for the results data and instead was fed it from the server via JSON in a script tag, that removes one request allowing for room for this new request. Some of the reasons this is a bad idea though include: response caching issues, timing measurements would need to filter out MiniProfiler differently to how its done now, more JS would be required for a more complicated UI display process.

Maybe it is best to approach this as a "How do we make MiniProfiler extensible to capture other client side perf-related details" and go from there? Might co-incidentally help #466 too.