w3c / csswg-drafts

CSS Working Group Editor Drafts
https://drafts.csswg.org/
Other
4.38k stars 643 forks source link

[css-view-transitions-2] Ignore offscreen elements from participating in transitions #8282

Closed khushalsagar closed 1 day ago

khushalsagar commented 1 year ago

Currently if an element has a non-none computed value for view-transition-name, it participates in the transition irrespective of whether it is in the visible viewport. This means the element will be rendered, which has significant computational and memory overhead, even if it is never seen by the user. If the developer wants to avoid this overhead, they have to keep track of the visibility of each element and only add view-transition-name to the onscreen ones.

The proposal to make this case easier is as follows:

See prior discussion on this here.

noamr commented 3 months ago

That seems ok. Although I'm not sure how you'd add the "but consider ink overflow" option in future. (I realise I'm not being a lot of help here)

I think in the future we can add a view-transition-inclusion-edge property that defaults to border-box and can be ink-box (or some such). Or we can have a view-transition-inclusion-margin that can take a px value, defaults to 0, and can be auto in the future which would be computed by ink overflow.

noamr commented 3 months ago

OTOH, I'm not sure if we want to extend this in the future to enable exclusive cutoff rather than inclusive, or even clipping (only the rect that's part of the viewport is snapshotted).

Perhaps this can be (extended in the future to) 4 properties, with a shorthand (replace the word cutoff with any other word that we choose):

view-transition-cutoff-area: viewport | root;
view-transition-cutoff-rule: inclusive | exclusive | clip; /* this is somewhat similar to canvas/svg fill-rule */
view-transition-cutoff-margin: length-percentage;
view-transition-cutoff-extents: border-box | paint-box | ...;

view-transition-cutoff: inclusive viewport; /* shorthand to include any element that intersects with the viewport */
view-transition-cutoff: inclusive root; /* this is the default */
view-transition-cutoff: clip viewport; /* this is the default for the root element*/
khushalsagar commented 3 months ago

For deciding whether an element's box is visible or not, we should probably use the snapshot containing block. Since any part of that block can become visible during the transition (URL bar hidden vs shown) so we want to capture an element if its in that block.

Re: which reference rect to use, I agree using ink overflow would be ideal but +1 to start off with the simpler approach of using the border-box given the complexity with ink overflow. There's probably more cases where authors would be ok with using the border box (ignore the element if only a small shadow is showing).

Anchor positioning is running into this too (see https://github.com/w3c/IntersectionObserver/issues/522). Hopefully having use-cases come up from both these features will give us the motivation to figure out exposing ink overflow too. @fantasai @tabatkins fyi.

noamr commented 3 months ago

For deciding whether an element's box is visible or not, we should probably use the snapshot containing block. Since any part of that block can become visible during the transition (URL bar hidden vs shown) so we want to capture an element if its in that block.

Absolutely

nt1m commented 3 months ago

I would prefer if the default value gave the UA flexibility to apply whatever behavior it judges more efficient. Ideally developers wouldn't need to touch this.

noamr commented 3 months ago

I would prefer if the default value gave the UA flexibility to apply whatever behavior it judges more efficient. Ideally developers wouldn't need to touch this.

That already (partially) exists, but this feature is more about allowing the author to use this for art-direction, like decide that if an element is offscreen in the old state it should/shouldn't fly in for the new state.

khushalsagar commented 3 months ago

I would prefer if the default value gave the UA flexibility to apply whatever behavior it judges more efficient.

The current spec technically doesn't. The browser has to render offscreen content because the animation might bring it onscreen. For example, you delete an item from the list and the next item slides onscreen into place while its content (item #) animates.

But we can allow the browser to ignore elements which are away from the viewport given a UA defined margin. That's likely what authors want anyway, otherwise its only the last few frames of the animation when the element comes onscreen which looks weird.

So this property can be a tri-state of: visible, hidden and auto where auto uses UA heuristic to decide whether an element is visible or hidden.

nt1m commented 3 months ago

I can see a use for hidden, but not sure about visible, auto should be smart enough to do the right thing given the UA knows about the start and end states. Don't really have a super strong opinion here though

khushalsagar commented 3 months ago

I can see a use for hidden, but not sure about visible

This naming choice wasn't the best. My thinking behind the tri-state was these modes:

nt1m commented 3 months ago

Capture if the element is onscreen. If it's offscreen, the UA can skip based on heuristics of how far offscreen it is.

I was thinking the UA could be smarter and detect if there would be any visible effect to omitting the animation based on the position of start+end state, but might not be trivial, which would make visible technically not useful. I'm fine either way.

khushalsagar commented 3 months ago

I was thinking the UA could be smarter and detect if there would be any visible effect to omitting the animation based on the position of start+end state

That's the fundamental problem which requires a web API here. The browser can't know what the end state will be when its caching the old DOM, since it doesn't know what the new DOM is yet.

nt1m commented 3 months ago

I was thinking the UA could be smarter and detect if there would be any visible effect to omitting the animation based on the position of start+end state

That's the fundamental problem which requires a web API here. The browser can't know what the end state will be when its caching the old DOM, since it doesn't know what the new DOM is yet.

Don't we have capture the new state steps where we collect newly added tagged elements?

khushalsagar commented 3 months ago

At this point in the algorithm when we cache the old state, we just know that the named element in the old DOM is offscreen. We don't yet know if its corresponding named element in the new DOM will be onscreen. And when we get to new state steps, the old DOM has already been replaced by the new DOM.

I could imagine an optimized implementation internally preserves a cheap copy of the old DOM; such that generating an image of the old DOM elements can be deferred until the browser knows what the new DOM looks like. And skip rendering/caching the image if the new state is offscreen without any web observable effects. Hmmm, can webkit do this? :)

jakearchibald commented 3 months ago

this feature is more about allowing the author to use this for art-direction

As an author, I want to +💯 this.

I don't think heuristics work here. By the time the browser knows the full animation for a group, it's already captured the old view. This feature is about being able to exclude the old view. Also, even if the animation won't intersect with the viewport at the start of the animation, there's no guarantee that will remain true - the DOM can change.

As an author, sometimes I'm happy with an element animating from outside the viewport to outside the viewport. Sometimes I'm happy for it to animate from its original position, outside the viewport, to its new position inside the viewport, or vice-versa. But sometimes I'm not. It depends on the thing being animated.

If that design choice is taken away from me, the feature is useless. This feature is about design choice.

khushalsagar commented 3 months ago

If that design choice is taken away from me, the feature is useless. This feature is about design choice.

That's a fair point Jake. There is no doubt that we need an API here because it's about the UI the author is trying to design which the browser can't infer. So a heuristic only solution is not feasible.

That said, we still need to decide what should be the reasonable default behaviour. I was assuming that an animation where both the start and end states (or just the one state for entry/exit transitions) of the DOM element are offscreen should be skipped, because the user will never see that animation. But I'm likely missing a case since you mentioned: "sometimes I'm happy with an element animating from outside the viewport to outside the viewport". Do you have an example in mind?

Since the browser can't know if the end state will be onscreen, we have 2 choices for the default:

  1. Conservatively always capture, which is the current spec. The pro is that it's likely to be more visually correct. The con is its suboptimal because we're spending resources for offscreen animations.

  2. Ignore elements in the old DOM if they are far offscreen which is inverse of above, trading visual correctness to err on better perf. My inkling is that authors are more likely to notice the visual glitch and fix it by using the property proposed here than to realize the performance footgun of offscreen animations.

What would be your preference for the default behaviour?

jakearchibald commented 3 months ago

That said, we still need to decide what should be the reasonable default behaviour. I was assuming that an animation where both the start and end states (or just the one state for entry/exit transitions) of the DOM element are offscreen should be skipped, because the user will never see that animation.

That isn't true in cases where the element will pass through the viewport.

But even in cases where the element won't not pass through the viewport, that might change in the next frame, or before the animation is complete.

I think the current behaviour is the right default. Same as CSS animations - we don't cancel out-of-viewport animations.

khushalsagar commented 3 months ago

I think the current behaviour is the right default. Same as CSS animations - we don't cancel out-of-viewport animations.

The fact that out-of-viewport CSS animations don't add rendering cost is a reason to consider a different default here. I especially feel that after seeing examples like https://github.com/w3c/csswg-drafts/issues/8320#issuecomment-2023077559. For an infinite scroller type of case, authors will add a name to all list items without realizing its causing the browser to render offscreen content. If they do need it to render (for cases you mentioned) and the browser doesn't by default, they'd notice a visually incorrect animation and opt-in to it.

But ok to take either approach at the WG meeting.

jakearchibald commented 3 months ago

You're always going to have the initial capture cost, because you need to do that before the 'end' position is given (but again, the end position can change throughout the animation).

I think that example is a misunderstanding of how to use view transitions, in the same way * { will-change: transform } is a misunderstanding. Authors should only be putting view transition names on things that need to transition.

If we want to allow for the case where developers give everything a view transition name, I think we need a different system altogether that eliminates the need for the initial capture in cases where it isn't needed.

noamr commented 3 months ago

Note that the spec does allow clipping the contents of elements to the snapshot containing block (while still logically capturing the element and creating pseudo-elements for it), see https://www.w3.org/TR/css-view-transitions-1/#capture-rendering-characteristics-algorithm.

khushalsagar commented 3 months ago

^ we only clip the root snapshot to the snapshot containing block. Non-root elements are captured in entirety (modulo hardware constraints). If we were to clip element snapshots to the snapshot containing block, they'd be completely empty for offscreen elements. IIUC the examples @jakearchibald is mentioning don't want that behaviour, since the animation can bring the content onscreen we need to capture the offscreen pixels.

I think that example is a misunderstanding of how to use view transitions, in the same way * { will-change: transform } is a misunderstanding. Authors should only be putting view transition names on things that need to transition.

That's fair but will-change comes with a huge disclaimer about its rendering characteristics. Since the property is explicitly about adding rendering cost for animation smoothness, authors have to realize what they're doing when they use this property. view-transition-name's primary purpose is element matching. The offscreen rendering is pretty subtle so I doubt authors will think about it. In fact most will probably assume what you said above, "same as CSS animations ...". Put an animation on everything you need and the browser will deal with optimizing offscreen stuff.

Picking a default to always capture for this new property is a bit like picking transform as the default value for will-change from a perf standpoint. I realize it's more nuanced since will-change is purely perf, while there is visual impact based on the default we use here.

jakearchibald commented 3 months ago

Let's say I want to perform a view transition where I reorder a list:

I would consider it a bug if the browser decides not to perform the animation I explicitly requested. It would be really buggy if the browser sometimes decides not to perform the animation I explicitly requested.

khushalsagar commented 3 months ago

It would be really buggy if the browser sometimes decides not to perform the animation I explicitly requested.

Ok, that I see now. It would indeed be very annoying if it sometimes works and differently across browsers. So let's not do any UA based margin heuristic.

Let's say we pick "only capture when onscreen" as the default. Would you say it would still feel buggy if you had to add the view-transition-inclusion-area property in addition to view-transition-name if your named element can be offscreen? Because then it's part of the API contract we're setting up with this property. Just the name is not enough to say, "I want this to always animate".

bramus commented 3 months ago

if you had to add the view-transition-inclusion-area property

Feels more like something to put as a descriptor in the @view-transition at-rule, as this applies to the VT as a whole instead of individual elements.

jakearchibald commented 3 months ago

Let's say we pick "only capture when onscreen" as the default.

It would break all "slide in/out from the top/side/bottom" transitions.

jakearchibald commented 3 months ago

I guess advice to developers would be: Browsers are about to break your transitions. Add * { view-transition-inclusion-area: everywhere } to prevent this 😄

khushalsagar commented 3 months ago

Feels more like something to put as a descriptor in the @view-transition at-rule, as this applies to the VT as a whole instead of individual elements.

Wouldn't authors have cases where this is different for different elements?

It would break all "slide in/out from the top/side/bottom" transitions.

Just to be clear, it won't break a slide in from out-of-viewport type entry (name only in new DOM) animation. If the element in the new DOM is onscreen, it'll get captured and you can make its pseudo slide in from out of viewport to its final state. Same thing for exit (name only in old DOM) animations.

But yea, the list example you gave where you swap the first and last entry, and the last entry is offscreen, would break.

I guess advice to developers would be: Browsers are about to break your transitions. Add * { view-transition-inclusion-area: everywhere } to prevent this 😄

Ha, well if authors end up using that CSS then changing the current behaviour is moot. I feel like you're convinced that authors will know to not put a view-transition-name unless that element needs to be animated, i.e., it will come onscreen during the transition. And changing the default here will just get in their way.

I'm worried that won't be the case, especially as we make it easier to declaratively add names with features like https://github.com/w3c/csswg-drafts/issues/8320. But I also don't have any other idea for avoiding this perf footgun...

So let's keep the current behaviour. Also easier from an implementor standpoint, no compat risk to deal with. :)

vmpstr commented 3 months ago

I think it's reasonable to consider making animations that begin and end off-screen not animate by default (and hear me out).

It is indeed going to become easier to mark things as participating in a view transition, via auto naming and auto matching. There are also thoughts about making something like transition: content where a view transition may implicitly start on an element when something about it changes.

All of these features would make it easy to have animations happening completely off-screen. And by easy I mean even without the author realizing it. To Khushal's point, this is going to likely regress performance more than a CSS animation would (although I would challenge that offscreen CSS animations are "free"). This is due to the fact that view transitions include visual captures.

If by default, we don't run an animation if the start and the end states are off-screen, then in currently Chromium implementation we would still run a capture, but we can get rid of it quickly. We can also see if we can figure out some implementation optimization for this, but I can't think of anything off top the of my head. In any case, we'd at least have a possibility of optimizing this.

What we lose is the list reorder case, where a thing intentionally flies through the viewport. That's ok? We can add a property that opts into the behavior of always capturing regardless of off-screen-ness

khushalsagar commented 3 months ago

@vmpstr trying to tease the API contact you're proposing. The browser is free to not generate pseudo-elements for a view-transition-name in the old DOM if:

For the case where a name is only in the new DOM, the browser can optimize out its rendering as usual since we're producing live snapshots.

Not generating pseudo-elements is the web observable difference here. So an author can't bring the pseudo-element onscreen only during the transition. Internally an engine can optimize by caching any representation of the old DOM until the new DOM is ready.

jakearchibald commented 3 months ago

Another case to consider is https://simple-set-demos.glitch.me/dust-no-raf/

// Dummy element for the dust
for (const name of ['dust-1', 'dust-2']) {
  const div = document.createElement("div");
  div.style.viewTransitionName = name;
  div.style.contain = "paint";
  document.body.append(div);  
}

These elements are created to produce groups that can be used within the transition, but they don't relate to any real content. They're only in the "old" view.

It feels like this would become unreliable under the proposed behaviour.

noamr commented 3 months ago

A potential perhaps-legit optimization the UA can do is this:

However, this type of optimization is a tangent to this discussion.

The thing I think we should solve here is - allowing the web developer to decide that for particular view-transition participants, if it's offscreen in one of the phases, the animation should act like an exit/entry transition. "I would prefer this article to fade in rather than to fly from the bottom if it comes from outside the viewport". The only way to achieve this today is by measuring/interscecting with javascript, and IMO it's an important enough use-case to make declarative.

bramus commented 3 months ago

A potential perhaps-legit optimization the UA can do is this: …

This feels like a bit too much hand wavy magic. I’d rather see authors explicitly opt-in to this, like they do with [loading=lazy] and content-visibility: auto;.

jakearchibald commented 3 months ago

The thing I think we should solve here is - allowing the web developer to decide that for particular view-transition participants, if it's offscreen in one of the phases, the animation should act like an exit/entry transition.

Yes, that's the feature developers are asking for, and the use-case that was identified in the previous issue to this one https://github.com/WICG/view-transitions/issues/84

noamr commented 3 months ago

A potential perhaps-legit optimization the UA can do is this: …

This feels like a bit too much hand wavy magic. I’d rather see authors explicitly opt-in to this, like they do with [loading=lazy] and content-visibility: auto;.

Agreed, using it as an example of (not too simple) things that implementations can do without changing the spec, which doesn't solve the problem presented in this issue.

khushalsagar commented 3 months ago

The thing I think we should solve here is - allowing the web developer to decide that for particular view-transition participants, if it's offscreen in one of the phases, the animation should act like an exit/entry transition.

That's fair. This is the primary use-case for this issue. That said, because this property is defining the behaviour for offscreen elements in View Transitions it intersects with the UA optimizing offscreen animations. Even if we don't change the default, we'd want authors to use this property to opt-out of caching offscreen content if they know the transition won't bring it onscreen.

It feels like this would become unreliable under the proposed behaviour.

This property can always be used to explicitly specify whether an element is captured irrespective of its viewport relative position. So the author can opt-in to any behaviour (which would reliably happen across browsers) if the default doesn't work for them.

Just to reiterate, my concern is limited to caching a representation of old DOM elements. As per spec, the author can bring an image of a named element in the old DOM in the viewport at any point. So the UA must generate and cache it for the lifetime of the transition. And while it's visually correct, I'm not sure if authors need that behaviour in most cases or understand the additional performance cost associated with it. There is no such issue with the new DOM snapshots because they come from a live element. The UA can already decide whether to paint them based on the position of the pseudo-element rendering that image.

noamr commented 3 months ago

Sorry to backtrack, but I think there is a problem we haven't considered with this, unrelated to the optimization options.

Everything is fine when the viewport stays where it is, but if we implement this suggestion it might produce undesired results when scrolling (or otherwise changing anything about the viewport).

Imagine the following scenario:

There are 3 options here:

  1. If we don't capture that element at all, it would potentially appear double! as part of capturing the old root, and as the new element
  2. If we don't capture the element, but also ignore it in the parent, the element would disappear immediately when scrolling, and the new element would re-appear
  3. we can capture both states, but add a hint that lets the author use a specific animation when flying from out-of-viewport. This way, the author can decide, for example, to fade out the old element and fade in the new one.

(2) is perhaps OK but feels less flexible. We can achieve (3) in a "soft" way, by using UA view-transition-classes, e.g.:

::view-transition-group(header.-ua-from-out-of-viewport) {
  animation: fade-in-instead-of-fly;
}
khushalsagar commented 3 months ago

The user scroll down during the animation

At what stage of the transition is this happening? For both the old and new DOM we'll compute the element's visibility only once when we're looking for elements with names. So a visibility change after that (from scrolling) won't impact whether the element is captured or the pseudo-DOM structure.

And can you clarify in what situation the UA would add "-ua-from-out-of-viewport". The idea of UA provided classes sounds reasonable, just want to make sure I get the details right.

noamr commented 3 months ago

The user scroll down during the animation

At what stage of the transition is this happening? For both the old and new DOM we'll compute the element's visibility only once when we're looking for elements with names. So a visibility change after that (from scrolling) won't impact whether the element is captured or the pseudo-DOM structure.

The old element is out of viewport when capturing the old state, and the usef scrolls to a position where it's in viewport. So we considered it invisible when capturing but that state changed.

And can you clarify in what situation the UA would add "-ua-from-out-of-viewport". The idea of UA provided classes sounds reasonable, just want to make sure I get the details right.

If at the time of capturing the old state the element is invisible, the pseudos would receive this vt class. The author can then decide how to animate it if at all.

khushalsagar commented 3 months ago

The old element is out of viewport when capturing the old state, and the usef scrolls to a position where it's in viewport. So we considered it invisible when capturing but that state changed.

So the view-transition-name is ignored in the old DOM but it is applied in the new DOM (when scrolling brought that element into the viewport). There won't be a ::view-transition-old pseudo for that name. If the name has an ancestor with a name 2 cases are possible:

There will be a ::view-transition-new for it though. Would the :only-child pseudo-class suffice to detect this state?

noamr commented 3 months ago

The old element is out of viewport when capturing the old state, and the usef scrolls to a position where it's in viewport. So we considered it invisible when capturing but that state changed.

So the view-transition-name is ignored in the old DOM but it is applied in the new DOM (when scrolling brought that element into the viewport). There won't be a ::view-transition-old pseudo for that name. If the name has an ancestor with a name 2 cases are possible:

  • The named ancestor is the root in which case the old content is not captured at all (because root snapshots clip to the viewport).

Right, I guess animations with captured roots anyway don't work well with scroll. But this element can be a child of a captured element other than root as well.

  • The named ancestor is not the root in which case the old content could be in this ancestor's snapshot.

There will be a ::view-transition-new for it though. Would the :only-child pseudo-class suffice to detect this state?

That would be option (2). you'd scroll down and the element would disappear immediately rather than fade out. This is ok but perhaps not always the desired outcome.

khushalsagar commented 1 month ago

A quick summary of the main points to resolve here:

  1. The primary purpose of the feature is to ignore names on elements which are offscreen, we need to decide which reference rectangles to use to define whether an element is offscreen. The proposed resolution is that an element is offscreen if its ink overflow rectangle doesn't intersect with the snapshot containing block. Given the resolution on https://github.com/w3c/csswg-drafts/issues/8649, ink overflow is sufficiently spec'd for this use-case.

  2. The second question is about the syntax. From the options discussed above, view-transition-visibility seems like a good fit (it's kinda like content-visibility). The values being visible and viewport, where viewport means the element's view-transition-name is ignored if it's offscreen (as defined in 1 above).

  3. The last question is what should be the default value for this property. The current behaviour is visible so changing it would come with a compat risk. That said, from a perf perspective viewport is better because it avoids rendering of offscreen content which most authors won't realize. If the group thinks this is the right thing to do, we can collect data on the usage to decide if the compat risk is acceptable.

jakearchibald commented 1 month ago

This feature provides a condition to capturing the element. I don't feel visibility communicates that. Some bad ideas:

khushalsagar commented 1 month ago

^ A couple of thoughts for the options:

bramus commented 1 month ago
  • Another reason I like view-transition-visibility is that this feels conceptually similar position-visibility. Specify how view transition name should behave based on the visibility of the element. So would become a consistent pattern for authors to understand.

+1 on drawing the parallel to position-visibility.

  • For the values, I like always as the default (as-in always capture). in-viewport is nice to say "only capture when in viewport".

What about scrollers with clipped overflow where you’d want to use this? In that case in-scrollport seems more universal? Or maybe in-view to add some wiggle room?

khushalsagar commented 1 month ago

What about scrollers with clipped overflow where you’d want to use this?

Good point! How about element-visible or visible? Again drawing inspiration from position-visibility's anchors-visible.

bramus commented 1 month ago

Not a fan of the term “visible” (it has many interpretations) but good to go for something that is consistent.

Another (inconsistently named) option that came to mind is not-clipped which clearly communicates how it should be interpreted.

ydaniv commented 1 month ago

How about view-transition-overflow with possible values like clip/visible? But then this will only refer to stuff inside/outside of viewport, not to elements inside viewprot but not visible.

khushalsagar commented 1 month ago

not to elements inside viewprot but not visible

which case were you thinking about? If the element has visibility: hidden. I was assuming that visibility here is limited to whether the element has been clipped by an ancestor. And it looks like anchor positioning is doing the same thing here.

ydaniv commented 1 month ago

I was assuming that visibility here is limited to whether the element has been clipped by an ancestor.

Yeah, that's more powerful than just clipping the view. Could also be cases where element is clipped using clip-path or not visible because of opacity/mask/content-visibility/etc. So saying something like "optimize everything" is even more powerful than ignoring "offscreen elements"

khushalsagar commented 1 month ago

Hmmm ok. I would push back against occlusion but the cases you outlined seem reasonable.

noamr commented 1 month ago

I want to present a counter-argument to where this is going. The new prop is a simple condition: "if the element is 100% outside of viewport during capture don't capture it".

This feels like it would work great as an optimization for the cases where the element is out of the viewport throughout the whole transition, and for art-direction when we want to avoid very far fly-ins. Perhaps that's sufficient and we may come to that conclusion! But a lot of cases may fall in between... e.g. avoiding a fly-in that is "quite far" but the element still intersects the viewport by a single pixel, or wanting to change the animation when an element flys in rather than not capture it at all.

It could be that a customizable margin property in addition would fix that, but something about the condition feels rigid, like a very specific solution to a particular problem that perhaps doesn't extend well to adjacent problems?

This is not an opposition to the proposed direction, but rather a call-out to check if this rigid condition we're proposing aligns well enough with real world use cases in such a way that would be useful enough to replace IntserctionObserver-based solutions in those cases.