[web-animations-2] controlling animation frame rate

[graouts]

Apple has started working on an experimental feature (off by default in Safari Technology Preview) to control the animation frame rate of animations, with an explainer detailing an initial proposal.

To summarize, we would like to allow authors to specify a frame rate for animations such that it would be possible to identify the relative impact of animations, some preferring to run at lower frame rates – and thus allowing to optimize the cadence at which the page is rendered – and some at higher frame rates for maximum visual impact.

Authors would specify a frame rate, either via a keyword (auto, low, high, highest) or using an explicit value.

This proposal would tie in nicely with the notion of a CustomEffect (see https://github.com/w3c/csswg-drafts/issues/6861) such that scripted animations could easily adjust their frame rate without changing their logic.

Cc @smfr

[birtles]

Thanks for this. I read through the explainer but I was bit unsure about exactly where in the rendering pipeline the change in frame rate is realized. Particularly this note:

Note: there may be some nuances here around the timing of transition and animation events; for example, if the end of an animation iteration for a 30fps animation falls close to a 4ms boundary, when is the animationiteration events fired?

I think this applies equally to things like resolving the finished Promise, or updating ResizeObserver observers too.

My understanding of the proposal is that we are suggesting modifying the update animations and send events procedure to partition the set of animations into separate buckets such that only a subset of animations might be updated each time? Is that right?

If so, I think it would require a shift in the way time values flow down the timing hierarchy. Potentially some animations are taking their current time value directly from the timeline whilst others are using a cached value? If so, perhaps the values returned by the API could look inconsistent?

Or would we always update all animations at the local maximum frame rate for a given timeline (i.e. the maximum frame rate for all animations attached to it) but only dirty style for those that need an update in this particular frame? That way at least all the values seems consistent from the API?

Incidentally, if we made the timeline the point where the frame rate was controlled I think we would avoid a lot of these issues. It's more clumsy, since you'd need to create a new DocumentTimeline instance (and the mapping to CSS might get a bit more awkward too) but at least the API would always be consistent and easy to understand.

[birtles]

Oh, and do we need to give any thought to how we might discourage content from slapping frameRate: 'highest' on everything "for performance"?

[coreh]

Nice, this is a really interesting addition, and possibly a good way to strike a balance between smooth animations and battery drain (e.g. by ads).

I'm trying to wrap my head around CustomEffect and how it may eventually “supersede” requestAnimationFrame(), at least for the animation use-case, and I have a couple of questions:

1) If I'm writing (for example) a game with WebGL, would I essentially have my main game update/render "loop" as a CustomEffect callback?

let lastTime = document.timeline.currentTime;

const animation = new Animation({ frameRate: 'high' });

animation.effect = new CustomEffect(() => {
  // figure out how much time has passed
  const time = document.timeline.currentTime;
  const deltaTime = time - lastTime;
  lastTime = time;

  // Update game objects
  tick(deltaTime);

  // Render scene with WebGL
  render();
}, Infinity);

animation.play();

2) I'm assuming the progress argument (omitted on the example above) is on a scale from 0 to 1, so for the case where the animation duration is infinite (common for games), it would always be 0. Is that correct?

3) If not considered already, would it make sense to add a second deltaTime parameter, so that it doesn't need to be manually calculated?

4) What is the intended difference between the 'high' and 'highest' string values? Would 'high' essentially behave as an alias for 60Hz even for devices that support 90, 120 or 144Hz? Would 'highest' always provide the actual highest, or would it still interact with, say, low power mode and other factors?

5) Could I use multiple animations on the document timeline to drive in-game animations, possibly at different framerates from the main game render "loop"? i.e. Is it possible to have a separate CustomEffect that only triggers some data change, but doesn't actually re-render anything, and then the data is consumed by the "main" CustomEffect? A usecase I see for this is performing a physics simulation at 60FPS or even 30FPS while rendering the viewport at 120FPS for AR/VR, or maybe conditionally applying reprojection at 'highest' framerate if the main game logic running at 'high' framerate is unable to keep up. This would likely require a mechanism for specifying the execution "ordering" of different CustomEffects

6) How does this framerate mechanism interact with APIs that are already temporarily throttled/coalesced in some form? (e.g. Pointer Events) Do they become throttled to the highest framerate being used in the page? Or are they completely unaffected?

Thanks in advance and sorry for the wall of questions 😅

[chenglou]

I'd love to know too! A few of our animations are driven by gestures, currently inside rAFs. We're wondering what the future solutions would be so that we can leverage ProMotion, since Safari's capping rAF at 60fps

[flackr]

Incidentally, if we made the timeline the point where the frame rate was controlled I think we would avoid a lot of these issues.

I agree, I think the timeline is the right place for the framerate control. It also implies all animations on the same timeline would be aligned to the same ticks (which is presumably what you would want in most cases), and answers a bunch of other questions.

It's more clumsy, since you'd need to create a new DocumentTimeline instance (and the mapping to CSS might get a bit more awkward too) but at least the API would always be consistent and easy to understand.

I think we could make the mapping to CSS work. E.g. this would probably be specifying a different animation-timeline set up to be a different rate. Maybe using an @timeline rule to set up a different global named timeline which ticks at a different rate.

I also think it would be nice if you could modify the default document timeline to tick at a different rate if you want to as a site set the rate of your animations by default.

[stubbornella]

Oh, and do we need to give any thought to how we might discourage content from slapping frameRate: 'highest' on everything "for performance"?

Any single page is usually made up of different teams work. It isn't hard to imagine those teams competing with each other for higher frame rates.

I also wonder if this is giving developer control of something the browser should handle for them. What if the developer specifies something that makes the user experience worse. What would the browser do in that case?

[flackr]

Re @stubbornella, on non-webkit browsers this is what we have today: every animation tries to be the full frame rate (on webkit, all main thread animations are 60Hz). So this API is letting developers tell the browsers which animations are actually important so the browser has more information to include in its scheduling decisions. Without this, I'm not sure how the browser would be able to tell whether a particular animation was more or less important than another.

[mattgperry]

@flackr the linked document in the OP suggests these use cases for low frame rate animations:

• Animation on a small element (e.g. progress meter)
• Background ambient animation
• Fade animations (opacity, color, some filters) where lower frame rates are less noticeable
• Animated content that uses a steps() timing function

It would be trivial to automatically bump at least these last two into a lower frame rate. IMO rather than exposing an API for this, which so far is pretty poorly defined, it would be better to spec out values/easings that should run at a lower frame rate via CSS/WAAPI and limit those to 60fps.

Edit: To elaborate on "so far poorly defined".

auto, low, high, highest. These terms have no grounding in human perception. As others have pointed out, what is high vs highest? What is low? If specced these should be defined at least as minimum values, for instance:

low: 30fps high: 60fps highest: max vsync

So on a 60fps display high and highest are the same, not 15/30/60.

[flackr]

• Fade animations (opacity, color, some filters) where lower frame rates are less noticeable
• Animated content that uses a steps() timing function

It would be trivial to automatically bump at least these last two into a lower frame rate. IMO rather than exposing an API for this, which so far is pretty poorly defined, it would be better to spec out values/easings that should run at a lower frame rate via CSS/WAAPI and limit those to 60fps.

Steps is in fact already optimizable, and there are some optimizations in place in blink for this. A few challenges we ran into working with developers to lower the framerate of some animations using steps:

• Developers need to calculate the number of steps themselves, this was hard in some cases (e.g. animation timing is often defined in a different place than the keyframes)
• The CSS animations api can't combine steps + non-linear easing. With web animations you can apply a steps animations-wide easing and have a other per-keyframe easings to do this.
• Technically, if a developer doesn't carefully coordinate the start times of multiple animations running using steps the browser is still supposed to tick them independently which means frame generation isn't technically optimizable.

I do worry about degrading experiences automatically (e.g. the opacity / filter cases) without some developer control

[flackr]

@mattgperry I agree that the named rates are poorly defined and pushed back on this in other forums. I'm not arguing for this exact proposal, just arguing the case for why having an animation timeline tick rate would be helpful, especially if you could modify the default document timeline rate, a developer could easily:

• Opt all animations on the page into a lower rate.
• Align all animations running at the same rate (fewer wakeups / frames needed to render)

[mattgperry]

I can see the argument for opting all animations on the page to a lower rate but I think it would be incomplete without a way of excepting some animations/JS callback loops (CustomEffect/rAF) that want to run in sync with higher fps paints or input, like scroll or touch.

In terms of the proposed names, it would make more sense to be able to define a targetFramerate directly, with values higher than the display simply capped at the display framerate. We never want to scale framerates for perceptual reasons and this more direct setting would make it obvious what's happening here.

Or, given that there will be a handful of framerates that people will commonly want to define, also allow some names with explicit, unambiguous mapping:

• film: 24
• low: 30
• high: 60
• fluid/vysnc/max: 120/144

[flackr]

I can see the argument for opting all animations on the page to a lower rate but I think it would be incomplete without a way of excepting some animations/JS callback loops (CustomEffect/rAF) that want to run in sync with higher fps paints or input, like scroll or touch.

I was thinking something like this:

const highPerformance = new DocumentTimeline({minInterval: 0}); // use this timeline for max frame rate.
document.timeline.minInterval = 1000 / 30; // Run default effects at 30fps.

Note: I've used interval between ticks as it is similar to setInterval and allows expressing unlimited / maximum easily - with 0 - rather than needing to specify some high framerate.

I'm not sure if you change the default document timeline rate though how we would express whether we want to limit or not limit inputs so maybe it needs to be more nuanced.