Feature Proposal: optimized images

[jpchase]

See the explainer.

[dbaron]

So I have a few thoughts on this proposal; I think it's probably better to put them in one place rather than distributing them across a bunch of issues in this repo, so I'm going to stick them here.

/cc @loonybear

• I'm a bit worried about both maximum-downscaling-image() and image-compression()'s relationship to responsive design. If an image is going to be scaled to different sizes depending on the size of the display (window, phone, etc.), it seems like it's pretty easy to get into a situation where an image meets the policy most but not all of the time. And that seems like it's going to produce bad results for users. Given that responsive design is generally considered good practice, I'm not sure what do to here.
• The section on legacy-image-formats() was a bit confusing until I realized the "Note:" was all of the first bullet point except for its first sentence. Some styling (or sticking in a blockquote) might help. Similar for the other similar sections for the other policies.
  • I'm also a bit worried about the syntax proposed in that note, since it's using things like "JPEG" and "TIFF" as identifiers for image formats rather than MIME types. This would require creating a registry of such identifiers. (Is it "JPEG" or "JPG", etc.?)
• I wonder if GIF actually belongs on the default list of modern image formats. But I also think making such a list is going to lead to some interesting arguments between browser vendors.
• My initial reaction is that the more valuable parts of the proposal are maximum-downscaling-image and image-compression (relative to legacy-image-formats)... but those are also the parts that I think are more problematic in their relationship to responsive design. So I guess that leaves me unconvinced that this is something worth pursuing in its current form.

[clelland]

I'm a bit worried about both maximum-downscaling-image() and image-compression()'s relationship to responsive design.

+1 for responsive design -- we shouldn't stop encouraging that.

image-compression is more about the relationship between the file size and the number of pixels of image data it represents. If you serve a 32x32 image, and those 1024 pixels are dwarfed by 2MB of EXIF data and multiple thumbnails, that's something that the policy can alert you to. I don't think that responsive design comes into play with that one.

maximum-downscaling-image definitely interacts with responsive design, though. That's one reason that it's not a strict "image-must-match-container-size-times-dpr" policy: we want to allow room for full-width images that have to render on different viewport sizes.

In that case, though, I don't think the right answer is to just always serve a single full-bleed 5000px-wide image to all devices. A combination of <img srcset> and media queries should give you enough that you can deliver the right image to any device and avoid trying to fit any particular image into a container where less than 1/4 of the pixels are actually rendered.

I'm also a bit worried about the syntax proposed in that note, since it's using things like "JPEG" and "TIFF" as identifiers for image formats rather than MIME types.

Agreed. MIME subtypes is probably the right domain for these identifiers.

But I also think making such a list is going to lead to some interesting arguments between browser vendors.

That would certainly be interesting :) Avoiding that is one of the reasons that there's a syntax proposed (which I want to write up as a separate issue) to allow the developer to decide which formats should be accepted. Browser vendors can have their own opinions, but at the end of the day the developer opting into this should have the final say over whether or not they need gif support.

[dbaron]

The ratio used for image-compression() is currently described as "no more than X times bigger than the image size (width * height) on the web page", which I read (maybe incorrectly) as being the size that was being displayed rather than the size in image pixels. It's at least straightforward to change it to being a ratio against the image's internal size in pixels, which would remove the concern about responsive design. maximum-downscaling-image() seems harder to fix, though.

[clelland]

Hmm.. it does say that; maybe I've misunderstood. I think that it makes more sense if it's just about information that's intrinsic to the image, and not about the markup or the device. @loonybear -- is it actually supposed to compare # of bytes to the # pixels renderered on the display?

[Malvoz]

In the explainer about legacy-image-formats:

For an element, if its src is one of the modern image formats, the image will be rendered correctly; otherwise the image will be rendered with inverted colors.

Does it actually look for image formats within the src attribute? What if you do not specify one? See automating resources w/ client hints.

I suppose you would look for content-type in the response header of the image? If this is obvious to you, excuse my uneducated comment.

[clelland]

The current implementation in Chrome does just that -- it checks the Content-Type header to see what format the image is. If a browser does content sniffing (checking the first few bytes of the file to try to guess the format), then that should probably be tested as well.

I wouldn't recommend using the src attribute, to look for file extensions, etc. Those are easily misused, or omitted, as you mention. (The explainer should probably be fixed to make that clearer, thanks :) )

[annevk]

In fact, browsers actively ignore Content-Type (except for image/svg+xml), so you want to test something labeled image/[non-modern] that is actually modern as well.

[Malvoz]

Could this include a policy for image decoding?

As per https://github.com/whatwg/html/issues/2806#issuecomment-396992463 it seems (confirmed: https://github.com/WICG/feature-policy/issues/193) lazyLoad is to be a feature policy to allow authors control of the default value, seems sensible decode should have a policy too?

[ojanvafai]

I agree that setting the default value of the decode attribute is a sensible feature policy. Filed https://github.com/WICG/feature-policy/issues/182 to track that.

[Malvoz]

@jpchase

The explainer file location has moved, mind updating the link to: https://github.com/WICG/feature-policy/blob/master/policies/optimized-images.md

[jpchase]

Link to explainer updated in first comment.

[triblondon]

The current implementation of unoptimized-images in chrome appears to offer no means of detecting the intervention. legacy-image-formats and oversized-images also suffer from this problem. I tried capturing the image into a canvas but I got the actual image, not the placeholder (which incidentally offers a crazy way for people to circumvent these policies).

I thought someone smarter than me might have come up with a test but there seems to be nothing in web platform tests for these policies. So, are they testable?

(Since this affects multiple policies I've also raised it as a top level issue: https://github.com/w3c/webappsec-feature-policy/issues/279)