"decode" attribute on <img>

[smfr]

Decoding of large images can block the main thread for hundreds of milliseconds or more, interrupting fluid animations and user interaction. Currently, there's no way for a web author to specify that they want an image to be decoded asynchronously, so there are scenarios where it is impossible to avoid UI stalls.

To solve this problem we propose an "async" attribute on image elements. This attribute is a hint to the UA that the author has requested asynchronous decoding. This implies that if the UA paints an image after the "load" event has fired, but before the image has been decoded, the UA is allowed to not paint the image (rather than block on decoding it).

To notify authors when a decoded image frame is available, we propose firing a new event, "ready", on the image element. This would allow authors who require a fully-decoded image, in content that is sensitive to UI stalls, to wait for the "ready" event before doing something that brings the image into view (such as a CSS transition).

Some images repeatedly decode frames, for example, animated GIFs. In addition, the UA can throw away the decoded frame for a still image, requiring a re-decode. In these cases, we propose that the "ready" event only fires once, the first time a frame is available for display.

ISSUES: "async" is currently used to imply async loading, and it's possible that we'd want to use it in this sense for images too. Maybe call the new attribute "asyncDecode" or something else.

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

[grorg]

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

We could maybe follow CSS animations/transitions here. Make a new "imageDecoded" or "imageReady" event (I suck at names) and fire it on the target that has the new CSS property.

I wonder if you'll ever want some parts of the element to have async images, and other parts to not. e.g. borders vs background. If so, it seems like it would have to go in the new image() function. (It already has tags for rtl and ltr).

[grorg]

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

I do think it is confusing that async here is a slightly different meaning from script (which is "Set this Boolean attribute to indicate that the browser should, if possible, execute the script asynchronously." )

Or does it?... Would you consider image decoding to be the equivalent of script executing?

[grorg]

Of course, the HTML5 specification has a more complete definition of script async.

[smfr]

Actually reading that, I think 'async' here is quite a good match.

[duanyao]

Isn't off main thread image decoding already possible in browsers without a spec change? E.g.: https://bugzilla.mozilla.org/show_bug.cgi?id=716140 https://bugs.webkit.org/show_bug.cgi?id=90375

Does "async" also means lazy download of the image file?

[smfr]

Isn't off main thread image decoding already possible Possible, yes, but the main thread may still have to block on the decoder thread if it is still decoding an image when the main thread wants to paint the image. If a UA chooses not to block, then the user gets a a blank or partial image even though the load event fired (I'm not sure what Gecko does, and Mac/iOS WebKit have used main thread decoding for the past few years).

This proposal allows an author to present images with a guarantee both that the main thread is never blocked on image decoding, and that the image is ready to display when the "ready" event fires.

Does "async" also means lazy download of the image file?

No. It's purely about decoding.

[duanyao]

Once a UA implemented off main thread decoding, there is little reason for them to keep the old behavior, so still no need to specify "async" in HTML. Currently there seems no UA which (intentionally) blocks layout/paint on image download/decoding.

As to "ready" event, I think the situation can be compicated, because UA might drop decoded images that are no longer visible (out of viewport or in background tabs) to save memory. For completeness, "unload" event may be also needed.

[smfr]

Currently there seems no UA which (intentionally) blocks layout/paint on image download/decoding.

First, we're not talking about downloading here. This proposal doesn't change anything about how image downloading works.

Second, Mac and iOS WebKit block painting on image decoding (I work on WebKit for Apple, BTW). That's the problem we're trying to address. If other UAs have solved this problem, I would be interested to hear how.

[duanyao]

I am confused because I observed that UAs may display images increamentally on a slow connection, so I think decoding is not blocking painting, at least in such situation.

So does "async" also gurantee that the image is not visible when the downloading/decoding is in progress?

[smfr]

UAs can decode and paint an image that is not fully downloaded, yes. It may appear partial or low-resolution in this state, but the decode itself can still take time (100ms or more). I think the "ready" event should fire when the first full-resolution frame is available.

[annevk]

@igrigorik @yoavweiss may have opinions.

I think the API should be <img>.ready returning a promise. No opinions on the attribute, though we might in the future also want something that influences when the image is downloaded (e.g., an indication to the user agent that it's fine to load the image later).

[domenic]

Let's set aside the bikeshedding on the name and style of the event/promise for now, and try to get some multi-implementer interest in the base feature of using a new attribute to control image decoding. I think the key question is:

Second, Mac and iOS WebKit block painting on image decoding (I work on WebKit for Apple, BTW). That's the problem we're trying to address. If other UAs have solved this problem, I would be interested to hear how.

For example, a UA could choose to simply make all images have this "async" behavior and draw placeholders. Presumably this might not be great for authors (why, exactly?), thus this proposal to make it opt-in, if I am understanding correctly. If we were to, in certain circumstances, flip the default to async, this is starting to sound intervention-ish; /cc @RByers @ojanvafai

Another aspect that this is related to is off-main-thread image decoding. Some of the recent work on ImageBitmap has enabled this to be done in JavaScript; see https://developers.google.com/web/updates/2016/03/createimagebitmap-in-chrome-50. But this requires awkward contortions because then you have to draw it onto a canvas, instead of using a simple <img>. I am unsure to what extend this async proposal overlaps---would this allow off-main-thread image decoding? Is it in some sense sugar over the ImageBitmap technique that makes it less weird? /cc @junov since he's our ImageBitmap guy, both for Chrome and for this spec.

My sense is that this area is of definite interest to Chrome, and as such they're good candidates for second-vendor interest. I've tried to pepper @-mentions of people who might know more above. Any thoughts on Mozilla or Edge?

[smfr]

For example, a UA could choose to simply make all images have this "async" behavior and draw placeholders. Presumably this might not be great for authors (why, exactly?)

Some authors need a guarantee that if an image is ready, it gets painted. An image popping in later is not acceptable in some kinds of content.

If we were to, in certain circumstances, flip the default to async

Interesting idea, but I don't think it would fly compat-wise.

I am unsure to what extend this async proposal overlaps---would this allow off-main-thread image decoding?

I'm proposing this now specifically because WebKit has been doing work on off-main-thread image decoding, for some subset of images (webkit.org/b/155322 and related). I don't think a UA would be required to do off main thread decoding to implement this (for example, they could trigger decodes in something like a requestIdleCallback), but pages would still suffer from UI stalls.

[domenic]

Some authors need a guarantee that if an image is ready, it gets painted. An image popping in later is not acceptable in some kinds of content.

I think I understand what you're saying, but can you check if the following expansion is correct?

As a web developer it seems like all of my images just pop in at random times when the browser has finished with them and is ready to show them. The difference between showing a placeholder while it's fetching and then janking while decoding and then painting, vs. showing a placeholder while fetching and decoding, and then painting, is almost unobservable to me.

The difference is in the specific case where I end up listening to the load event. For example, if I create an image out of document, then wait for the load event, I anticipate that when I insert it into the document it will be painted synchronously. (This might even affect from-script-observable things like offsetWidth.)

But as an author, it seems like any time I'm not listening for the load event and assuming the image is fully ready at that time, I'd be able to sprinkle async on the <img> elements, and there would be no observable difference. Right?

[smfr]

... anticipate that when I insert it into the document it will be painted synchronously

That's exactly it. This is mostly useful for images created in script.

any time I'm not listening for the load event and assuming the image is fully ready at that time, I'd be able to sprinkle async on the elements, and there would be no observable difference

Yes. The UA may choose to do async decoding for some or all elements. Addition of the "async" attribute to an may be treated as a hint or may do nothing.

[domenic]

OK, cool, glad I understand!

What do you think of an alternate proposal, to delay the load event until decoding is finished, in all cases?

[smfr]

What do you think of an alternate proposal, to delay the load event until decoding is finished, in all cases?

You can't do that, because loading and decoding are two separate steps. Not all images that are loaded are decoded (we never decode those that don't get painted).

Note that fetching metadata from an image, to get its size, for example, doesn't count as decoding.

[ojanvafai]

I'm pretty sure Chrome strongly supports doing something roughly like this. I'm consulting with folks to make sure I say something representative of our opinion since we've had a lot of complicated discussion about this issue. Will report back soon (please ping me if I lose track of it).

[igrigorik]

Naive question: how would "ready" event work in cases where we have hardware decoding?

[smfr]

I don't think it matters. Hardware decoding happens under a software call to the decoder, just like software decoding. It just might be faster.

[junov]

Responding to @domenic 's earlier comment that there is overlap with ImageBitmap... Yes and no.

I've experimented with getting jank-free scrolling and canvas draws under intense image loading. I used XHR -> creatImageBitmap -> canvas. It worked like a charm, but it has one non-negligible drawback: it pins all the decoded images in RAM unless the app explicitly discards them, so implementing something like Facebook's infinite scrolling on top of this is a bit involved. It would require the web app to be responsible for evicting and triggering predictive redecodes in order to avoid extreme memory bloat (and OOM crashes). Considering that JS code has no visibility into the system's memory contention, using ImageBitmap as a blanket solution for de-janking image-intensive pages is really not that great.

This proposal lets the UA continue to manage the memory occupied by decoded resources, which is a big win for many use cases.

On the other hand, the ImageBitmap approach guarantees that the image will be drawn and that it will be fast. This is useful for cases that don't jive with this statement of the proposal: "if the UA paints an image after the "load" event has fired, but before the image has been decoded, the UA is allowed to not paint the image (rather than block on decoding it)"

[smfr]

If the author really want the image pixels to be available on first paint, they need to either insert the image in script after receiving the "ready" callback, or toggle some CSS so that the image becomes visible.

However, there's a problem with my proposal; there's no way for the author to indicate that the UA should start decoding an image (if that image is unparented), and I don't think UAs will want to start decoding all unparented images with "async".

[domenic]

However, there's a problem with my proposal; there's no way for the author to indicate that the UA should start decoding an image (if that image is unparented), and I don't think UAs will want to start decoding all unparented images with "async".

Since this is most useful for images created in script anyway, maybe an imperative "decode me now" API? Like

img.ensureDecoded().then(() => {
  // OK, now you can insert it into the document
});

Maybe "ensureReady" if we want to hide the implementation details a little more.

[smfr]

That could work, yes. I think that means you don't need an "async" attribute for images created in JS.

It could still be useful as a hint on content images to say that you're OK with an image temporarily painting blank even after the load event fires, trading off against possible UI stalls. No doubt people will want CSS pseudo classes to style the various states of an image.

[vmpstr]

As @ojanvafai mentioned, here are our (Chrome) thoughts after having discussed with our team:

We agree that this is an important area to look into and we’re excited to work on solutions in this space!

We see two important use cases with different characteristics. (1) Deferring: Ability to defer decodes of images in dom order to avoid janking the page. (2) Predecoding: Ability to know when images have been decoded so they can be inserted into dom without jank

An async attribute on images solves deferring, but not predecoding. An on ready event solves predecoding, but not deferring. It seems like these could potentially be solved separately.

We have some concerns about tricky parts of the ready event: * Does every image get "on ready" or just ones marked async? * Would a browser need to predecode all out of dom images so that they would receive this event? Or just ones marked async? * Is there an "on unready" event? * If an image is evicted and then re-decoded, is there a second ready event? The original proposal said it’s just the first time, but we’re worried this is too confusing and can lead to jank in subsequent uses. * If an author has a series of images (e.g. carousel) out of dom, how does an author communicate which ones are important to decode first?

Here's two rough proposals to handle each of these use cases.

(1) Proposal for handling deferring images: Give the async attribute three values: async, auto, and sync. sync behaves as today’s image elements do without any attribute specified, where once an image has been loaded it will appear immediately (and be decoded synchronously) if inserted into the document. Images marked async can get loaded/decoded best-effort without janking. auto would involve browser heuristics to decide if the image could and should be loaded async or if it needs to be sync. async is therefore just a more aggressive version of auto in practice. sync is mostly just there as a safety valve for developers in case browser heuristics get it wrong.

There’s no ready event here, as it’s not part of this use case. In particular, this helps avoid confusion around what happens when a decoded image is discarded and then later decoded again (e.g. because the image was scrolled out of view and then back in). The browser can treat it the same way regardless of whether it was the first time the image was being decoded or not.

We’re not 100% sure we’ll need the async value without more experience on trying to ship changes here. It might be that we can make all images that we want to async and just have auto and sync in the end.

(2) Proposal for predecoding images (similar to what @domenic suggested): Provide an explicit decode function for an image with some sort of callback when the image is decoded. This would notify the author that it's ready to be appended to the page without jank. The browser would promise to keep that image decoded for at least that raf frame. It would also allow an author to call this decode function again in the future if needed, and to prioritize images in an image carousel case. Finally, we might want to make this decode function return some success/failure (in case of an overloaded cache).

Possible sample code: var img = new Image(); img.src = ‘...’; // The decoded image is guaranteed to be kept in memory until the frame // after the decode promise resolves. img.decode().then(function() { // This is guaranteed to paint the image without flicker or decoding jank. document.body.appendChild(img); });

Other side questions: * What about large images that are scaled and cropped? Do these parameters need to be specified? We found that scaling and gpu uploading can be a significant part of image preparation time. Do we just punt on this? * How is image data that isn’t image elements handled, e.g. should we add an async keyword for background images? Divorcing async from decode makes it easier to do something consistent between image elements and CSS background images.

[smfr]

I like this analysis.

(1) Proposal for handling deferring images ... Images marked async can get loaded/decoded best-effort without janking

Presumably the effects of "async" on a content image is that it can show blank after its load event has fired, right? Seems fine if so. "auto" is weird as an attribute; I would prefer that we say that "async" is a hint and the UA is free to ignore it; if it becomes a cargo cult, I don't want to have to respect it on thousands of images in a document.

(2) Proposal for predecoding images I like the decode() Promise form; it makes it easy to figure out the behavior with repeated calls on the same image.

What about large images that are scaled and cropped? Do these parameters need to be specified? We found that scaling and gpu uploading can be a significant part of image preparation time. Do we just punt on this?

I say punt; doing scaling/cropping is usually a paint-time operation, and if you tried to do something for these, you'd end up having to do lots of things in the painting pipeline.

• How is image data that isn’t image elements handled, e.g. should we add an async keyword for background images?

Since you can't detect load on CSS images directly, it seems less important to support "async" for them.

[vmpstr]

Presumably the effects of "async" on a content image is that it can show blank after its load event has fired, right? Seems fine if so. "auto" is weird as an attribute; I would prefer that we say that "async" is a hint and the UA is free to ignore it; if it becomes a cargo cult, I don't want to have to respect it on thousands of images in a document.

To clarify a bit, the proposal is to have a single attribute called async, which could have three possible values (async/sync/auto). I agree that these values should still be just hints to the UA and it can ignore them.

The idea behind auto value and how it's different from async is that it allows the UA to adjust its heuristics to either include or exclude more images from being async based on whatever criteria it decides. In other words, async value would mean it's always safe to show a blank image after the load event fired; auto would mean that the UA should decide whether or not it is OK do this.

It's not entirely clear that we need both auto and async, but if auto is the default it would allow the UA to make these decisions for content that didn't explicitly specify what it wants.

[domenic]

Circling back to this to try to get a resolution, since we have a couple implementers definitely interested in implementing.

It sounds like there are now two distinct but complementary proposals in flight:

• A content attribute hint, tentatively async="", for the deferring use case. This would be used declaratively most of the time.
• An imperative API, tentatively decode(), for the predecoding use case. That would be used imperatively.

Let's use thread to work through the design of the content attribute, and use https://github.com/whatwg/html/issues/2037 for the decode() API.

It seems like the biggest outstanding point for the attribute to decide on is whether this is a two- or three-state attribute. Does anyone want to help further drive the discussion on that?

[domenic]

@smfr I see that in Safari Tech Preview 40 WebKit shipped an async attribute. Do you have plans to standardize that? It does seem nice and simple.

I'm curious whether @vmpstr still thinks we need a tri-state attribute, or the two-state one you implemented.

[smfr]

We would like to standardize "async", yes. As you say, it's very simple; it's just a hint from the author to the UA that the author is OK if this image is decoded asynchronously, and therefore might not be painted for one or more frames after the load event fires. We decided not to do any kind of "ready" event.

We found it impossible to do async image decoding by default in many scenarios where we wanted to without causing unwanted flashing, so this attribute allows authors to give the UA some more leeway.

[vmpstr]

We're finding similar issues with async images: it's hard to come up with a heuristic that allows us to do this without any unwanted flashes, so I think we should standardize the async attribute.

What are your thoughts on making it tri-state? Basically, it would have three states:

• "auto", the default state, would mean that UA can attempt to do async decoding in cases where it determines that there won't be flashes. This would help with long tail of web content that has large images but isn't trying to do anything fancy that can cause undesired flashes.
• "async", which means that always go async without checking heuristics. This is the async implemented by WebKit
• "sync", which means don't do async decoding even if the heuristics seem to indicate that it would be ok.

The reason we're leaning towards this is basically so that UAs still have some leeway of using a heuristic to help the performance on sites that aren't addressing large image problems on their end. At the same time, it gives the tools to the authors that do want fine grained control.

From the implementation perspective, UAs could just adopt auto = sync, which means that for WebKit the code changes would only need to change the interpretation of the attribute.

[othermaciej]

I don't think we want the "sync" state, just "async" and "auto", which is sufficiently covered by a binary attribute. The attribute is a hint, so the UA is free to apply a heuristic when "async" is not specified, at least as I see it. You can force sync-like behavior but without causing stuttering by using Image.decode(), so I'm not sure we need an additional way to force it. What's the use case you envision for explicitly forcing "sync" instead of accepting the default "auto" state?

[vmpstr]

For me it's just a safety knob in case that we do end up doing async heuristic where the user really didn't want it. You're right however that the img.decode should cover those cases.

Ok, I think for the sake of simplicity, I'm sold on just having a binary state.

[chrishtr]

The use case for the "sync" state is it's a very simple way for developers to opt out of any heuristics, without having to use the image decode API or write any extra script.

The image decode API also only works in a simple way the first time the image is decoded and inserted into the DOM. It is much more awkward to attempt to use it later in time during the web page's lifecycle, which may lead to situations such as the image being scrolled offscreen, falling out of the cache, then returning.

A "sync" attribute on the image is a clean way to handle all such situations, and also makes the sync/async image attribute self-contained without an implicit dependency on another API.

[othermaciej]

The use case for the "sync" state is it's a very simple way for developers to opt out of any heuristics

Do we have evidence that there's a need to opt out of heuristics in a simple way? With the introduction of "async", heuristics can lean conservative and we can gradually convince content authors to use async.

The image decode API also only works in a simple way the first time the image is decoded and inserted into the DOM. It is much more awkward to attempt to use it later in time during the web page's lifecycle, which may lead to situations such as the image being scrolled offscreen, falling out of the cache, then returning.

I don't think that's the case. You can always call decode() again and wait for the promise to resolve. Why is that awkward?

[chrishtr]

The use case for the "sync" state is it's a very simple way for developers to opt out of any heuristics

Do we have evidence that there's a need to opt out of heuristics in a simple way? With the introduction of "async", heuristics can lean conservative and we can gradually convince content authors to use async.

Simple is better than not-simple. I anticipate browsers becoming more aggressive with heuristics over time to address the image decode jank problem. A simple opt-out solution for sites which fall into corner cases of those heuristics means it will be easier to implement those opt-outs and therefore more sites will do it. It also allows a fully declarative approach, rather than the image decode API, which requires managing the image outside of HTML and then inserting/showing it after decode.

The image decode API also only works in a simple way the first time the image is decoded and inserted into the DOM. It is much more awkward to attempt to use it later in time during the web page's lifecycle, which may lead to situations such as the image being scrolled offscreen, falling out of the cache, then returning.

I don't think that's the case. You can always call decode() again and wait for the promise to resolve. Why is that awkward?

When do you call decode? For the scroll use case, this requires adding a listener to see when the image might be coming near the screen again, which requires an IntersectionObserver or custom code. It also forces the developer to assume that the image fell out of the decode cache and call decode(), even though it might be useless.

[othermaciej]

When do you call decode? For the scroll use case, this requires adding a listener to see when the image might be coming near the screen again, which requires an IntersectionObserver or custom code. It also forces the developer to assume that the image fell out of the decode cache and call decode(), even though it might be useless.

If the site is paging parts of the DOM in and out, it will be doing all this anyway. If the image is in the DOM the whole time and just scrolled out of view, it's up to the browser to arrange to have it decoded in a timely fashion. I don't think we should give websites a way to force scroll jank for previously decoded images. I'm strongly against adding a new web platform feature that would let web content authors force bad scrolling.

[chrishtr]

If the site is paging parts of the DOM in and out, it will be doing all this anyway. If the image is in the DOM the whole time and just scrolled out of view, it's up to the browser to arrange to have it decoded in a timely fashion. I don't think we should give websites a way to force scroll jank for previously decoded images. I'm strongly against adding a new web platform feature that would let web content authors force bad scrolling.

I definitely agree that we should not add a feature which requires scroll jank. However, "sync" does not require scroll jank, it just requires displaying all content of a tile at the same time. If the browser is not done with decode and raster for a tile, it will checkerboard that tile (behavior which is already present in the composited-scrolling paths in Chrome and Safari, and I assume all browsers) but scrolling will still proceed, possibly displaying checker-boarded tiles for a brief period.

A second point: the image decode API does not allow the developer to express that two images must display at the same time as surrounding content, because each of them has a different promise with its own lifecycle. It doesn't suffice to wait for both promises, because it is only guaranteed to be valid for a short period. OTOH, placing "sync" on both expresses exactly this, which I expect is a common use-case from developers.

[othermaciej]

Are there websites that benefit from whole tiles being blank when it's slow to decode an image while scrolling, instead of the image being blank?

For the two-image case, maybe we should enhance the decode() API to specify a list if anyone actually has this use case. I don't think we want to encourage developers to insert multiple images into the DOM with forced sync decoding specified since that risks jank. If the "both images or nothing" case is really important, we should provide a way to do it without risking either flashing or jank.

Can you name specific examples of websites that would want either of these behaviors? It's hard to argue with the expectation that it's a common use case without concrete examples.

[chrishtr]

Are there websites that benefit from whole tiles being blank when it's slow to decode an image while scrolling, instead of the image being blank?

All websites currently receive this behavior. It's hard to say which would prefer the current behavior to a new one. I'm pretty sure many will not like the new behavior. People don't like visual distractions.

For the two-image case, maybe we should enhance the decode() API to specify a list if anyone actually has this use case. I don't think we want to encourage developers to insert multiple images into the DOM with forced sync decoding specified since that risks jank. If the "both images or nothing" case is really important, we should provide a way to do it without risking either flashing or jank.

Adding a more advanced feature to the decode() API strikes me as much more complicated than simply supporting a "sync" attribute.

Can you name specific examples of websites that would want either of these behaviors? It's hard to argue with the expectation that it's a common use case without concrete examples.

There are lots of top sites that try really hard to avoid extra blinking or visual distraction during load or animations. Professional, advanced sites often do things like fading in after load, to make the UX more elegant, not to mention native apps, which often tightly control when images display. Many people also commented on the launch of Edge that its loading seemed faster and smoother than Chrome's because it didn't pop in content during load as much. Both of these are examples of showing the equivalent of checkerboarded tiles rather than partial content.

One might argue that my anecdotes are mostly about load, but I think they demonstrate that it's common to want all content to display at the same time, and I think the "sync" attribute is a clear and easy way to achieve that, without requiring interaction jank.

[smfr]

I definitely agree that we should not add a feature which requires scroll jank. However, "sync" does not require scroll jank, it just requires displaying all content of a tile at the same time

This isn't true in the situations where browser still falls back to main-thread scrolling (e.g., in Safari, overflow scroll, iframe scrolling, and some scenarios with background-attachment:fixed), so "async=sync" [weird combination!] would still cause main thread stalls.

[smfr]

I do worry about cargo-cultism here. If we provide "sync", it might be seen as a big hammer to make image loading "better", defeating browser attempts to get more stuff off the main thread.

[chrishtr]

This isn't true in the situations where browser still falls back to main-thread scrolling (e.g., in Safari, overflow scroll, iframe scrolling, and some scenarios with background-attachment:fixed),

Good point.

But: Chrome aims to have composited scrolling on all scrolls over time (by implementing raster-inducing scrolling, which would be able to mutate the display list on the compositor thread and re-raster tiles). IOW the user agent has ways to make this not blocking user interaction for scroll.

I should also point out another reason why I think "sync" is a good idea: not only does it allow developers to specify sync behavior in cases where they want it, but it also allows user agents to have a freer hand to be a bit more aggressive with deferring images in "auto" mode, because developers will have an opt-out. I think this will make it easier for user agents to reduce jank on the long tail of content which uses default settings. It sounds like Safari's early implementation experience matches that of Chrome's recent work - it's hard to come up with heuristics that don't introduce blinking when sites don't want it.

so "async=sync" [weird combination!] would still cause main thread stalls.

Yeah, the name would have to be changed in this case. :)

[othermaciej]

I should also point out another reason why I think "sync" is a good idea: not only does it allow developers to specify sync behavior in cases where they want it, but it also allows user agents to have a freer hand to be a bit more aggressive with deferring images in "auto" mode, because developers will have an opt-out.

I don't think that's right. Having an explicit opt-out doesn't really make it more ok for a heuristic to break existing content. At least that's not the strategy we take.

[chrishtr]

I don't think that's right. Having an explicit opt-out doesn't really make it more ok for a heuristic to break existing content. At least that's not the strategy we take.

Could you explain why you think so? Any heuristic will have false positives.

[vmpstr]

We've had a long discussion about this and here's why we feel that the attribute should be tri-state (all of this has been more or less mentioned throughout this thread):

One of the objectives of this attribute is to give the developers a tool to give the browser a hint that the image should be asynchronous (async value). This is a hint that makes it possible to apply some set of aggressive heuristics in order to minimize jank. I still suspect that we'll have some heuristics, since it may be that small images, for example, are quicker to just synchronously decode rather that rasterizing content without the image first and then rasterizing it with the image after.

The default state would then seem to imply that we should effectively do synchronous decoding. Or have a very conservative heuristic that when applied and the image is deferred, the effect is indistinguishable from synchronous decoding.

However, we would still like to try and improve the user experience on sites that may not adopt the async attribute. The approach that would be nice is to launch the async attribute as described above and then over time improve on the "default" heuristic to be more aggressive.

With this approach, it is highly desirable to have a "sync" state which would revert back to the conservative heuristic initially used for the default state. Note that sync doesn't mean we have to do the decode synchronously. It's a hint meaning that the user prefers this image is synchronous (ie it's important to display with other content). As long as the effect is indistinguishable from sync decoding, the UA is free to do whatever it needs to.

So, on the spectrum of heuristics from conservative to aggressive, we would then have this:

• sync means conservative heuristics
• async means aggressive heuristics
• default/auto means somewhere between the two.

To emphasize again, this doesn't mean that the user can force the browser into a jank. It just means that the user prefers content to not be deferred. As we would spec this, the language would rely on saying "should" more than "must" when dictating the behavior of the browser.

With this, of course, naming the attribute itself "async" is a bit confusing, but we could discuss the name of the attribute if we can agree on the tri-state value.

[othermaciej]

@vmpstr You're using "the user" a lot to refer to what I think is "the content author". Could we keep those distinct? The end user is actually a relevant party here and we should not conflate the content author's preferences with the user's preferences.

On the substance of the argument: if the three values select one of three browser-specific heuristics, which may change over time even within a single browser, then I don't think authors will be able to use it correctly. If anything, this would exacerbate @smfr 's worry that the attribute will start getting used in a cargo cult manner, so it will stop being useful signal.

@chrishtr It's true that any heuristic has false positives, and you have to tune false positives vs false negatives. But I don't think an explicit opt out makes it ok to have more false positives. So it doesn't materially change this tradeoff. I can't imagine resolving a bug that says a heuristic is too aggressive by telling the reporter that it's the site's fault front adding the opt-out. Especially since (per @vmpstr ) it's not even a hard opt-out, it's just a hint to the heuristic.

Given all this, I'm still in favor of only two states. I think the third state is likely to create more problems than it solves.

[chrishtr]

On the substance of the argument: if the three values select one of three browser-specific heuristics, which may change over time even within a single browser, then I don't think authors will be able to use it correctly. If anything, this would exacerbate @smfr 's worry that the attribute will start getting used in a cargo cult manner, so it will stop being useful signal.

"sync" means "I prefer to have it all display together, please do that". That's not a heuristic, right? Regarding @vmpstr's comment about it not being required, that language is proposed to give the user agent the right to override in situations where the UX is really bad, i.e. an intervention, which is not normal behavior.

"auto" leaves it up to the browser and is the default. This is the heuristic which would evolve over time as we optimize for existing content.

"async" means "I prefer to display content fast, and not block on this image"; it's expected the browsers will implement this fully async (on a background thread, with invalidation at the end), so it's not a heuristic either. (I assume something like this is what WebKit has implemented? I'm actually curious, don't really know.)

@chrishtr It's true that any heuristic has false positives, and you have to tune false positives vs false negatives. But I don't think an explicit opt out makes it ok to have more false positives. So it doesn't materially change this tradeoff. I can't imagine resolving a bug that says a heuristic is too aggressive by telling the reporter that it's the site's fault front adding the opt-out.

I don't see why we shouldn't resolve a bug sometimes as "please use sync if you really want the image to display at the same time as other content". Sometimes developers have to take action to keep their sites working well in new browsers.

If we could re-design images fully, I think probably it would have been a two-state of sync or async. But since "auto" is the existing web, and some browsers implement sync only currently (Chrome and Safari shipping versions) or some async-like heuristic (Edge and Firefox), we have to do something to move to a future state where jank is reduced. I think there are developer use-cases for sync and async, and also "don't know/don't care". In any case, legacy content exists and it would be a real shame to be wedged into a state where we can't do anything about it without breaking developer use-cases without simple workarounds.

[smfr]

Sometimes developers have to take action to keep their sites working well in new browsers.

I think we try really hard to avoid this; can you cite another example where the user experience on legacy web content degrades as browsers change?

[chrishtr]

I think we try really hard to avoid this; can you cite another example where the user experience on legacy web content degrades as browsers change?

Blink changes behavior regularly to improve interop between browsers and remove footguns through the Intent process. We do so only when the percentage of broken content appears to be very small and the severity of the break is also small (though there are exceptions to even this for changes we deem critical, such as removal of window.showModalDialog). We regularly change DOM APIs in ways that are known to cause JS errors on a small number of sites.

We also have been making changes explicitly targeted at reducing footguns and jank. At times these are phrased as an "Intervention" where we believe that improving UX for users outweighs fidelity to the current status of an API. A good recent example related to scrolling is forcing event listeners that are not explicitly marked as non-passive to be considered passive. See here for more details: https://www.chromestatus.com/features#passive. All shipped changes are also listed under /features there.

In the case of image decoding, the changes we would make would only make images blink somewhat more often on some existing sites, but otherwise the site would be perfectly usable (and faster).