Detection time window may not allow enough URL Metrics to be collected

[westonruter]

Originally in https://github.com/WordPress/performance/issues/1494#issuecomment-2305825246.

I discovered on my blog that I had a very old URL Metric from before the XPath indices were fixed to be 1-based instead of 0-based (#1191). Looking at the URL Metric data, I see the timestamp is 1712354502.752218 which is for a day back in April. It appears that URL Metrics are not being collected frequently enough, or my site is truly devoid of traffic. I think for my site it may be in part due to page caching, where the detection time window (od_detection_time_window) is too short, so it is only collecting metrics for visitors who happen to visit when there is a MISS to the page cache, and if there is a stale-while-revalidate caching strategy going on with Varnish then it is even more likely for the visitor to never be within the detection time window (currentTime - serveTime > detectionTimeWindow):

https://github.com/WordPress/performance/blob/f8dad3497f6240e2975733c3b277e91e35b254a6/plugins/optimization-detective/detect.js#L179-L187

I think this logic needs to be revisited.

[felixarntz]

Related is a conversation @westonruter and I had at WordCamp US: To reliably handle dynamic behavior on the client-side, the behavior needs to be controlled by the client-side, to account for full page caches which will lead to most requests not to reach the PHP logic. This basically means two things:

• We need to inject the script, or at least a very minimal version of the script, unconditionally, because we can't know from the server-side whether the current request needs the detection to run (e.g. if it never hits the PHP process).
• We need to retrieve data that is dynamic in another way, it is not reliable to inline it with the script. Some data that is specific to the current URL (e.g. urlMetricsSlug) or a general configuration setting (e.g. storageLockTTL) is probably okay to inline, but most other data (e.g. urlMetricsGroupStatuses) will get out of date when a full page cache is used. Such data would probably need to come from a REST endpoint instead.

Defining the right solution for this probably goes beyond this issue, maybe we should open a new one, that is more broadly about how to support full page caches in a way that the Optimization Detective functionality is still reliable? I think prioritizing this will be crucial to get broader adoption, as lots of sites use a full page cache which will affect the behavior negatively - sometimes minor, sometimes more severely, depending on the cache configuration.