Baseline frame for comparison

[paulirish]

The calculations of SpeedIndex in WPT are based on video frames, which are captured at 30-60fps during load. However, we're using frames which are only captured when the browser ships 'em. In practice, this nets out one particular key difference:

There's no guarantee we'll have white/blank frames at the start of our trace.

As a result, we can use a contentful frame as our baseline, which leads to an incorrect representation of 0% visual progress. (I've profiled a number of sites whose first frame shipped is already rather complete)

It feels like the correct solution here is to generate a white frame at the timeOrigin to use as our first/baseline frame. +@pahammad is this the right approach?

Somewhat related to #21

[pahammad]

If you look into the details of VisualMetrics GitHub repository, Pat's sampling schema for computing SpeedIndex is not uniform either. This issue is subtle but important. There's a simple way to fix this. Instead of using the original proposed Speed Index equation (from time-0 to visualComplete), break up the compute into two parts. So, instead of SI = sum (from 0 to VC) {1-VC%}, you can do SI = firstPaint + sum(fP to VC){1-VC%}. The first contentful frame generally corresponds very well to firstPaint marker, which is already available through synthetic measurement toolbox, and this provides a consistent lower bound to SI. Both the sums (or integrals) I wrote are equivalent, but the second one eliminates the SI=0 situations. Hope this helps. I wanted to write a blog post about it one of these days, but didn't get around to it yet.

[pahammad]

By the way, our implementation of PSI (perceptual speed index) in the VisualMetrics GitHub repository already accounts for this two part computation and handles this empty frame problem. You can look at the loop where PSI is updated for details.

[pahammad]

In the equation: firstPaint + sum(fP to VC){1-VC%}, firstPaint needs to be in milliseconds.

The bayesian explanation is the following: when the user is staring at white/blank screen, the cost of delay is purely measured in the waiting time. Once there's something to see (starting at firstPaint), the cost is the area above the curve for the visual completion curve (as explained in the SI equation).

[pahammad]

Let me tag +@pmeenan, just so he can add his 2 cents.

[pmeenan]

Depends on what you are trying to measure. In all cases you want to make sure the first frame you have is the same as the user's starting point.

In the canonical case of loading a page from nothing, you want to make sure your first frame is a blank white window.

In the case of a page-to-page navigation, the starting frame should match the screen that the user is navigating from (though, there are frequently cases in testing where users will manually insert a white frame by blanking out the document before starting the next navigation). Mostly because "start render" will get flagged as the point the browser blanks the page (commit) and not when the content loads but in the case of SPA's or in-page actions without navigations you only want the deltas.

The current logic is looking for relative changes between the start state and end state to explicitly handle the multi-step sequence cases.

[paulirish]

Thanks to both of you. These comments are invaluable.

Fixed in #30