search

eggpi / mc886

Sketches and auxiliary files for a machine learning class
1 stars 1 forks source link

hit rate vs PLT (or speedindex) metric #2

Open mcmanus opened 10 years ago

mcmanus commented 10 years ago

very cool stuff. it certainly is nice to have a property greater than just the page/subresource relationship.

The only thing I would be concerned about is that time to render (WPT speedindex, basically) has a large element of long-pole-in-the-tent to it.. so statements like:

"An interesting remark to be made here is that, while we don't expect all subresources for a given page to fall on the same cluster, we do expect them to group together to some extent -- [..] the chosen cluster usually contains over 60% of the subresources."

basically, if you're not batting a really high percentage its possible you won't see a lot of improvement in the end result at all - especially if critical subresources such as js or css are the ones that miss the boat... if that's the case it doesn't matter that images were moved quickly.

and 60% strikes me as kinda low.. maybe I'm just naive there?

So I'm really psyched about this - but lets make sure we measure with the metric that we really care about. (speedindex, wpt, shaped networks).

eggpi commented 10 years ago

Hmm, that's a very valid point -- I wasn't really aware of that long-pole-in-the-tent behavior, but it sounds like something that should be addressed in the model. Can we identify those critical subresources somehow? Or are we just considering js and css to always be critical? I'm considering expanding the model a little bit with one or maybe two extra features for each subresource, and some measure of importance derived from that behavior sounds like a great thing to use.

I'm not too familiar with how WPT/PLT is being used to track Necko performance (I assume you're talking about Stone Ridge?), I'd love to hear more about it. But yes, I certainly agree those are metrics we should consider.

About the 60% thing, I only meant to point out that the features I'm using right now do capture some of the "closeness" between subresources we know should be close together -- that is, they're not just scattering points randomly all over the graph. But it's definitely not to be taken as a measure of the overall success of the model, just as an indication that using these features can get us somewhere.

eggpi commented 10 years ago

Just a note: we might have all the information we need in the timestamps for resource loads -- the farther a resource's timestamp is from the timestamp of the page that loads it (in the same load, of course), the more it delays page load.