Indication of density for correct intrinsic size?

[kornelski]

If user embeds an image with <img src=""> without explicit size and uses CH to send "Retina" image, the image won't be in higher-resolution, it will remain lower-resolution and be displayed at twice the size.

Should there be a feature that allows the server to indicate that it has used CH and sent higher-density image, so that the browser knows correct size of the image?

[yoavweiss]

That issue also impacts srcset and picture (Actually, this is the way I checked which resource was downloaded in WebKit/Blink's LayoutTests.). I agree that this can impact unsuspecting authors that fetch their images from a CH-enabled 3rd party (unlike srcset/picture, where this issue is predictable).

Actually, an opt-in mechanism (discussed at #8 ) will mitigate that issue, since authors opting-in will indicate their support for HD images (which have that issue by definition), and will take care of it, by specifying image dimensions.

[kornelski]

edit: srcset shouldn't be affected. If "2x" image becomes larger, that's an implementation bug. Good implementation shouldn't make image bigger:

Each img element has a last selected source, which must initially be null, and a current pixel density, which must initially be undefined. When an img element has a current pixel density that is not 1.0, the element's image data must be treated as if its resolution, in device pixels per CSS pixels, was the current pixel density. For example, if the current pixel density is 3.125, that means that there are 300 device pixels per CSS inch, and thus if the image data is 300x600, it has an intrinsic dimension of 96 CSS pixels by 192 CSS pixels.

So if I understand above correctly, then <img srcset="100x100.png 2x"> should display at 50x50px and would be half the size of <img src="100x100.png">.

I agree that this problem affects <picture>, since mere media query doesn't define the density. I've fixed that in my <pic> proposal.

[igrigorik]

Should there be a feature that allows the server to indicate that it has used CH and sent higher-density image, so that the browser knows correct size of the image?

That's Vary: CH. If that header is missing, then that's a signal that the server didn't do anything with the CH hint.

[kornelski]

Vary tells client that it did something with the header, but doesn't tell what, so the client still doesn't know whether image is 1x or 2x.

[igrigorik]

Correct. I'm assuming we're talking about a DPR only hint at the moment - that's a limitation of Vary.

Key is the answer for multiple variables: http://tools.ietf.org/html/draft-grigorik-http-client-hints-00#section-2.5

[kornelski]

Is Key supposed to affect intrinsic size of the image (CSS/page layout)?

[igrigorik]

Given that it's not implemented in any browser today, the answer is "no". Whether it should, once it's available, is a separate story.

[kornelski]

So I'm asking for the "separate story" — to be clear, I'm not talking about caching here. I'm concerned about CSS layout problems caused by sending 2x images without ability to tell browser to display them at 2x. CH covers sending part, but doesn't cover displaying at 2x, so you get 2x displayed at 1x and breakage.

Use case:

I have an existing non-trivial website. The site uses Backbone to dynamically update some views that contain images, there's a <canvas> that renders a graph with icons.

I'd like to install mod_pagespeed or proxy via CDN that automatically (wherever possible) makes images high-resolution on "Retina" displays.

If intrinsic image dimensions are correct, then to implement that you need:

• sniff CH on image requests and send higher-resolution images with some indication of their DPI. Done!

If intrinsic image dimensions are not correct — as in currently implemented CH — then:

• modify HTML markup to add width/height to <img> — easy for static HTML (although you get latency hit if you do it on the fly), but how would you handle markup dynamically created by JS applications/templates?

• modify CSS to add background-size — possible in most cases, but cascade makes it hard. Given such markup:

  html {background-size: 100%} html.blue-theme {background-image:url(blue.png);}

if you modify .blue-theme:

html.blue-theme {background-image:url(blue.png); background-size: 20px 30px}

then you'll override cascade with value author did not intend. OTOH if you don't set background-size then all CSS sprites will be screwed up.

• modify code that uses <canvas> — pretty tough. You'd have to change prototype of Context.drawImage() to do ctx.scale(0.5); drawImage(); ctx.scale(2.0) around drawing calls of rewritten images, and be careful not to change draws of canvases/video/imagedata.

Overall it's probably mostly-tracktable problem, but need to add explicit sizes everywhere makes FEO quite invasive and fragile.

And it's even more complicated than simply using srcset and image-set(). With just these — no CH needed:

• rewrite static markup to add srcset
• rewrite image url() in CSS to image-set() — both can be done on-the-fly since you don't need to know image size at the time when you send the markup.
• if canvas used <img> from the document, then it'd just work. Otherwise JS would safely fall back to 1x (or you could apply the same tricks you'd for CH for complete coverage).

[yoavweiss]

@kornel - Talking to Ted O'Connor, srcset display dimensions should be impacted by the x qualifier chosen. I'm not sure it's currently implemented that way, but I will test it.

So, to go back on my previous claim, srcset doesn't suffer from this issue. (or at least it shouldn't)

[igrigorik]

A couple of ideas that come to mind:

(1) If CH is enabled via explicit opt-in (see #8), then as Yoav pointed out, one strategy would be simply to require that the site author provides the right width and height attributes in their markup - in practice, this also has the performance benefit of much faster layout, and this is something we recommend as best practice for FEO solutions already as it enables us to resize the image to appropriate dimensions (before scaling for DPR). The downside is that it does mean you have to be judicious about specifying dimensions... I don't think this is entirely unreasonable, but admittedly, not optimal either.

(2) Because the asset selection occurs on the server, the server needs to indicate which asset was selected - e.g. the client may have sent a 2.0 hint, but the server can still serve a 1.0 asset due to configuration policy or other logic it may implement. In this case, perhaps a couple of ways to handle it:

• (a) Client sends CH-DRP: 2.0, server responds with asset and DPR: 2.0 header which indicates the actual DPR of the served asset. In other words, the server needs to signal the DPR of chosen asset. The client then should use this information to automatically calculate the appropriate display size.
• (b) Instead of serving an HTTP header, the DPR can be embedded in the meta-data header of the served file. This is more risky, since some intermediate optimization proxies may explicitly strip this data.

(2a) feels like the best mechanism. Would love to hear any other thoughts, proposals, or ideas on how this could be addressed (or if it needs to).

[kornelski]

IMHO definitely (1) is not good. Of course specifying dimensions is a best practice. I'm always trying to do that, but on large websites I find it's not always possible to do in 100% of cases: you always have some images from some CMS you can't easily control, some odd template you forgot about, or image URLs is variable. Plus if you don't have server-side preprocessor that can automatically parse HTML and update the dimensions, it's annoying.

However, that gives me an idea: how about reporting dimensions of <img width/height> (and maybe background-size?) to the server?

A common performance mistake is to embed huge image in a downscaled <img>. If there was client-hint for displayed size, then server-side automation could automatically resize the image to dimensions specified in the markup (the dimensions would only be taken from <img> attributes — they're available to the preload scanner before layout is done).

And if width/height was reported, then the server would know which images can be safely upgraded to 2x size. That would enable FEO services to automatically and mostly-safely do DPR negotiation, and skip those which could end up being displayed at wrong size.

(2b) might be a good idea too. It's going to be hard to introduce it, but long-term it may be the right thing: imagine if you could save Retina image from Photoshop and have it "just work" in every browser and image viewer. Maybe it's not too late to make that a WebP feature?

[igrigorik]

Reporting width/height is a good idea, but it's also an entirely different subset of the problem we're discussing here -- see #7 (and with its own gotchas). Let's keep this topic focused on intrinsic size.

[kornelski]

There's an overlap. Since Client-Hints DRP negotiation doesn't work only for images without dimensions specified in the markup, telling the server when image has dimensions specified would help servers limit DPR negotiation only to safe(ish) cases.

[igrigorik]

Sure, I guess that's another option: only send DPR header when width/height is present -- call it option (3). Albeit, personally, while it mitigates some of the raised risks, it also seems to be one of the least compelling.

[igrigorik]

Still a work in progress, but a first attempt at "server confirmation" language: https://github.com/igrigorik/http-client-hints/blob/bc5822a29a21c0486879b691bf3a0d320eb8d083/draft.md#server-hint-confirmation

In theory, that should be sufficient to address the issue we're discussing here. Having said that, still need to vet the implementation complexity and risks -- will report with what I find out.

[guypod]

Nit picking: should the response header include a prefix? If not, how come the response header doesn't need a prefix while the request header does?

[igrigorik]

@guypod semantics. The CH- prefix indicates that it's a hint (client > server), whereas the response header advertises a property of the resource (e.g. DPR). Incidentally, this is also why I think we need the CH- prefix..

[igrigorik]

FYI.. http://crbug.com/303863

[igrigorik]

Added "DPR" selection confirmation header in v1 draft: https://github.com/igrigorik/http-client-hints/blob/master/draft-grigorik-http-client-hints-01.txt#L255

Also check the example section below it. If you have any feedback, feel free to reopen.

P.S. Christian is close to landing support for it in Chrome.