search

cubiq / scrollbench.js

A browser scrolling benchmark
BSD 3-Clause "New" or "Revised" License
6 stars 1 forks source link

Rationalize FPS and dropped frame measurements #6

Open wiltzius opened 11 years ago

wiltzius commented 11 years ago

www.adidas.com/com/apps/snowboarding/

On my Windows machine, registers as "248/250" dropped frames on Chrome. Similar on Mac. The FPS counter reports ~20 FPS throughout, altohugh there are some longer hitches. But certainly it made more than 2 frames....

wiltzius commented 11 years ago

See also inception-explained.com

On Windows Chrome reports a steady 59FPS, and it looks like it works at about that rate, but the benchmark reports 698/700 dropped frames and an average 20FPS.

cubiq commented 11 years ago

could you report the total number of frames and the total time? also your PC config.

On my MBP I get 90 dropped over 240, that is high but reasonable for the amount of elements that are scrolling.

The FPS value is hard to cheat, it's just a proportion between the travel and the time. Also there might be some scripts in those sites interfering with SB (maybe they even use RAF?), but unless we find a way to async scroll by default, this will often be the case.

wiltzius commented 11 years ago

Focusing on just one example, http://inception-explained.com/ on a Retina MBP reports 696/698 frames dropped. This is on Chrome Beta on OSX.

cubiq commented 11 years ago

Is the refresh rate standard (60hz) on MBP? I believe there's an option in the preferences to check this.

Also would you report total time and travel as well?

cubiq commented 11 years ago

made some more tests. On my PC on inception-explained.com I get the following results:

Time: 13.184 Total frames: 698

This is approx a timer of 18.9ms. To get a 100% smooth transaction you'd need a result of at least 16.667 and in fact the result is 345 dropped frames. Even if the animation seems smooth, it is not. We can lower the dropped frames threshold to get better results, but if we want to stick on 60fps the math seems legit.

This is of course true only for RAF and 60hz monitors.

natduca commented 11 years ago

Should we just not report dropped frame count? It may be a bad metric, as in uncomputable without the refresh rate.

On Mar 11, 2013, at 4:26 AM, Matteo Spinelli notifications@github.com wrote:

made some more tests. On my PC on inception-explained.com I get the following results:

Time: 13.184 Total frames: 698

This is approx a timer of 18.9ms. To get a 100% smooth transaction you'd need a result of at least 16.667 and in fact the result is 345 dropped frames. Even if the animation seems smooth, it is not. We can lower the dropped frames threshold to get better results, but if we want to stick on 60fps the math seems legit.

This is of course true for RAF and 60hz monitors.

— Reply to this email directly or view it on GitHubhttps://github.com/cubiq/scrollbench.js/issues/6#issuecomment-14708092 .

wiltzius commented 11 years ago

Perhaps I don't follow this correctly. How is it computing the dropped frame count now? Taking 60Hz * timed duration of scroll = denominator, and number of rAF callbacks are the numerator? Or...?

Using "long frames" might make sense (report number of frames that took longer than ~18 ms based on the time difference between rAF timestamps)?

cubiq commented 11 years ago

we check every frame with the previous one. If the difference is greater than 55, bam! dropped frame! :)

wiltzius commented 11 years ago

OK, so we time each frame compared to the last one's timestamp, and if the difference is greater than 55 (ms?) then we count that as a dropped frame.

This is confusing, though, because then if all of the frames are over that threshold we'll report 100% dropped frames, despite the fact that we actually do produce some.

Matteo you mentioned you're now computing FPS frame by frame instead of averaged totals... how does that come into play here?

cubiq commented 11 years ago

55 are fps or 18.18ms.

We are probably using the wrong term here. With RAF it's hard to know if a frame is really dropped, we would need a deeper HW API to know that.

We can just guess that if a frame is computed at less than 55fps than that frame is dropped. It may happen that the scroller is so slow that RAF can't catch up and we skip/miss a refresh (usually set at 60hz). So in this sense we can call it "dropped".

The update I did today makes the FPS value 99% reliable. Would be nice to test it against 50hz or 120hz monitors.

wiltzius commented 11 years ago

Renaming this issue to get more at what it's really about.

I have a Windows machine with Firefox on it that I'm looking at now that can go in either 50Hz or 60Hz mode. That's extra-confusing, so let's just consider 60Hz for now (some of the numbers I was reporting earlier were from 50Hz mode, which made them look even more off).

Even at 60Hz I can now get 61.6 FPS or higher in the reported results, with high dropped frame counts (e.g. on the facebook news feed)

It seems like should not be possible to measure FPS above 60Hz on a 60Hz display when measuring with RAF...?

wiltzius commented 11 years ago

How did the update to make the FPS value more reliable work?

Since as you point out we don't know the true refresh rate of the monitor, it seems like what we want to list are slow frames, not dropped frames. That metric and a rational FPS metric should put us in better shape?

cubiq commented 11 years ago

RAF is not a rock solid 60hz clock :) @natduca knows better than me, anyway you rarely get a steady flow with it.

when the browser is warmed up and the webpage doesn't have heavy elements RAF goes actually FTL (or at least this is how it is reported). Consider that 61.6FPS is 16.36ms instead of 16.667ms, so it's really a small difference. To that you may probably also add some errors caused by the timer.

If you have 100 frames, 50 at 55fps and 50 at 62fps you get 50% dropped frames (that is bad) but an average fps of 58.5 (that is good). The average FPS is important but it must be accompanied by the number of slow frames. That gives you an overall vision of how the animation went.

I'm now comparing every single frame timestamp with the previous one. Before I was just making the FPS value out of the totals. The new result should be a very precise average.

wiltzius commented 11 years ago

Yes, that makes sense.

Reporting the number of slow frames (i.e. frames that look longer than some threshold) and calling them that instead of dropped frames, plus an averaged FPS metric, sounds good. Otherwise I think its too confusing that you can have most frames reported as "dropped" when in fact the browser is making frames.

So returning to the 50Hz monitor case is my final question: on Chrome it seems to line up about right -- get top out at 50FPS measurements both with our built-in FPS counter and with scrollbench's report. However on e.g. Firefox it reports higher numbers than when the monitor is in 60Hz mode. Perhaps Firefox's rAF implementation is based on a timer?

cubiq commented 11 years ago

I've never been able to get solid results out of firefox rAF.

Regarding dropped frames, we can call them "slow frames" but we have to define an arbitrary threshold anyway. We can take the FPS parameter as an avg and report only frames under that average, but what if all frames are slow and the average is say 20fps? We would have 0 slow frames, 20fps...

the best would be a graph maybe? with time on X and fps on Y... but it's a hell lot of data for each test (especially long pages)

cubiq commented 11 years ago

please try the latest version. I've removed the "dropped frames" and introduced the "steadiness" parameter.

It tells you if the frame rate is constant.

ps: you may need to refresh the report iframe to see updates

natduca commented 11 years ago

Steadiness is really promising. How do we compute steadiness?

On Thu, Mar 14, 2013 at 10:03 AM, Matteo Spinelli notifications@github.comwrote:

please try the latest version. I've removed the "dropped frames" and introduced the "steadiness" parameter.

It tells you if the frame rate is constant.

  • If the FPS is high and the steadiness is good then you have a smooth animation.
  • If you have high FPS but low steadiness it means that the animation is jumpy.
  • If you have low FPS and good steadiness it means that animation has a constant pace but the end result is not smooth
  • If FPS is low and steadiness is poor it means that you have a jumpy and rough animation

ps: you may need to refresh the report iframe to see updates

— Reply to this email directly or view it on GitHubhttps://github.com/cubiq/scrollbench.js/issues/6#issuecomment-14915020 .

cubiq commented 11 years ago
natduca commented 11 years ago

Love it. Can we put that in a mouse hover or something like that on the results iframe?

On Mar 14, 2013, at 4:00 PM, Matteo Spinelli notifications@github.com wrote:

— Reply to this email directly or view it on GitHubhttps://github.com/cubiq/scrollbench.js/issues/6#issuecomment-14934740 .

cubiq commented 11 years ago

I've added a description in the report page when you click on the value.

wiltzius commented 11 years ago

This seems good, although I'll want to see how it works on a few more pages (I tried it on a couple and it seems to make sense). I think we need to explain a little bit more explicitly how we're measuring, but that's just a wording thing.

What are the options for the steadiness reported value? "poor" "good" and....? I can write some copy.

wiltzius commented 11 years ago

Having just run the benchmark on ~6 sites in IE9 to test there, I definitely think we need to expose a little more explanation.

It isn't clear what "jagged" means in the steadiness box. I assume these are the count of frames >10% above or below the average frame duration? If we call this something more obvious (or explain in the text) it'll help. I think we should also explicitly list how many frames were above vs. below the time.

So we'd say something like:

Steadiness: (poor|good|??) Jerky frames: XX / YY

And on click you get: Steadiness tells you if the framerate is constant. Ideally you want a steady framerate AND a high FPS for smooth animation. The steadiness value can be good or poor. Scrollbench counts the number of frames that are 10% longer or 10% shorter than the average frame duration over the course of the scroll. The number of frames of length >10% from the average are counted as "jerky frames". If more than 10% of frames are jerky, the steadiness value is poor. In this run X frames were >10% faster than average, Y frames were 10% slower, and the average frame time among the Z was ZZms.

Again I don't think this is about measuring methodology, which I like, it's just about making it explainable.

Thoughts?

cubiq commented 11 years ago

I agree it's a little bit cryptic right now. Steadiness can be poor/mediocre/good. You very rarely get a mediocre steadiness, though.

From a smoothness standpoint it makes no difference if a frame is faster or slower than average. The graph should be a flat line (hence the "jagged" term, but I like "jerky" more).

Anyway maybe we could simply create a third index that sums up steadiness and FPS and gives the actual overall smoothness of the animation (or we could call it ScrollBench Index)

cubiq commented 11 years ago

I've added the "overall" value to the report. It's an "average" between Steadiness and FPS. I made it from 1 to 5 stars. Click the value to get a description.