hdrSupported - Screen.video or simply Screen?

[chcunningham]

In #118 we proposed putting this bool on Screen.video, alongside other properties that would describe the video "plane" on devices with different video vs graphics capabilities. In #133 we're reconsidering the resolution pieces, leaving just the bool. Lets make sure we're putting this bool in the right place. Questions:

1. Up until now the discussion of the 2 plane problem has been about resolution. @lhalliwell @jpiesing, can you confirm whether we should expect HDR support to also be an issue here (e.g. only supported in video plane?)

2. Where does the 2-plane problem live? Is it the Screen or is it within the UA software? Is "video" a property of Screens? Thought experiment: say you have an HDR display connected to a laptop running a hypothetical browser which supports HDR for both images and video. Now say you disconnect the laptop and instead plug in a Chrome Cast dongle, which only supports HDR for video plane (assuming answer no. 1 is "yes"). Note: the Screen didn't change.

On the one hand, I think Screen.video.hdrSupported would function fine surface any video-vs-graphics plane limitations. On the other hand, it isn't conceptually very clean. Please discuss :).

PS - apologies for not raising in the call earlier today. my questions here are brand new since this afternoon.

[jpiesing]

1. Up until now the discussion of the 2 plane problem has been about resolution. @lhalliwell @jpiesing, can you confirm whether we should expect HDR support to also be an issue here (e.g. only supported in video plane?)

@chcunningham There are probably exceptions but, as far as I'm aware, the graphics system/plane will still operate in sRGB (i.e. SDR) even when video is HDR. When graphics and video are combined, graphics may be converted to HDR or both graphics and HDR converted to something that fits the panel.

[lhalliwell]

@chcunningham There are probably exceptions but, as far as I'm aware, the graphics system/plane will still operate in sRGB (i.e. SDR) even when video is HDR. When graphics and video are combined, graphics may be converted to HDR or both graphics and HDR converted to something that fits the panel.

Right. As long as there are no HDR image formats or CSS support, it's presumably wasteful for a browser to render at anything besides SDR, and no need for a page to query for this.

I didn't read the full proposal but for some existing devices (like CC Ultra), a boolean "hdr supported" isn't enough. Such devices can't do any significant colour space conversions, so the video's colour metadata is passed through to the TV. This in turn means the page needs to be able to detect what the TV supports. From what I've seen, newer hardware seems to have better ability to convert between colour spaces, so this problem may go away over time.

[vi-dot-cpp]

@lhalliwell I didn't read the full proposal but for some existing devices (like CC Ultra), a boolean "hdr supported" isn't enough. Such devices can't do any significant colour space conversions, so the video's colour metadata is passed through to the TV.

@chcunningham has addressed this concern in this comment on #118 by noting that MC.decodingInfo will be used to discern the TV's capabilities.

@jpiesing There are probably exceptions but, as far as I'm aware, the graphics system/plane will still operate in sRGB (i.e. SDR) even when video is HDR. When graphics and video are combined, graphics may be converted to HDR or both graphics and HDR converted to something that fits the panel.

To make sure I understand, are you saying there are differences with regards to hdrSupport between a TV's image plane vs. its video plane?

[jpiesing]

To make sure I understand, are you saying there are differences with regards to hdrSupport between a TV's image plane vs. its video plane?

@vi-dot-cpp Yes.

[vi-dot-cpp]

@chcunningham Where does the 2-plane problem live? Is it the Screen or is it within the UA software? Is "video" a property of Screens? Thought experiment: say you have an HDR display connected to a laptop running a hypothetical browser which supports HDR for both images and video. Now say you disconnect the laptop and instead plug in a Chrome Cast dongle, which only supports HDR for video plane (assuming answer no. 1 is "yes"). Note: the Screen didn't change.

I think your thought experiment is pointing out that HDR support for graphics changed even though the TV remained constant -- suggesting that the graphics v. video aspect aspect of hdrSupported is not tied to Screen. Not knowing much about cast devices, may I ask whether this is, or will/can be, the case?

Alternatively, could a TV only support HDR for either video or graphics @jpiesing @lhalliwell? In which case, there would be a need for Screen to disambiguate hdrSupported.

[jpiesing]

What does it mean to support HDR for graphics when web graphics are sRGB? Isn't that always SDR by definition or am I confused? If sRGB is SDR by definition then HDR support for graphics would be a meaningless concept.

[chcunningham]

I think your thought experiment is pointing out that HDR support for graphics changed even though the TV remained constant -- suggesting that the graphics v. video aspect aspect of hdrSupported is not tied to Screen.

Agree.

Not knowing much about cast devices, may I ask whether this is, or will/can be, the case?

In retrospect, I could have simplified this a bit to make the same point. Take any non-ultra cast (no HDR for either video or graphics), any laptop w/ UA supporting HDR, and any HDR screen. Switch between laptop and cast - only laptop can render HDR while screen remains the same.

What does it mean to support HDR for graphics when web graphics are sRGB?

The HDR plan for graphics isn't settled yet. Agree its historically sRGB. Discussion is active (Color on the Web CG discussed at tpac 2019 - having trouble finding the minutes).

With decodingInfo(), we are offering an API to say what kinds of videos we can render, and we're facing this snag about doing HDR video, but not HDR graphics. Maybe its fair to expect whatever HDR graphics future shapes up will include some similar APIs to detect support for rendering HDR graphics (i.e. that it isn't a given simply because Screen.hdrSupported == true)? Then cast/smart tvs could reply false for graphics in some TBD manner.

@hober - can you help connect us with some expertise on Screen spec? This feels like a TAG type API question.

[chrisn]

The HDR plan for graphics isn't settled yet. Agree its historically sRGB. Discussion is active (Color on the Web CG discussed at tpac 2019 - having trouble finding the minutes).

The minutes from the Color on the Web CG meeting at TPAC are here. See also this email from @svgeesus. There's a draft report that surveys HDR and wide gamut colour support across various web platform APIs. (I drafted the report a while ago, so the section on Media Capabilities lags the current discussion here.)

[vi-dot-cpp]

@chcunningham In retrospect, I could have simplified this a bit to make the same point. Take any non-ultra cast (no HDR for either video or graphics), any laptop w/ UA supporting HDR, and any HDR screen. Switch between laptop and cast - only laptop can render HDR while screen remains the same.

Why is the onus not on MC.decodingInfo() to report the HDR capabilities of the UA, which is the non-ultra cast device in this example. If the example were an ultra cast and any laptop w/o UA HDR support, MC.decodingInfo(), as opposed to hdrSupported boolean, would be used to discern HDR support of the laptop's UA.

Moreover, is the hdrSupported boolean useful for casting scenarios. Per your comment on #118, hdrSupported is redundant for cast, because MC.decodingInfo() checks the screen's properties.

[gregwfreedman]

i think of screen.video.hdrSupported as not simply a property of the panel, but a property of the panel as seen by the UA. in the case of Ultra Chromecast, screen.video.hdrSupported is obviously true. in the case of non-Ultra Chromecast, decoding.info might return true for HDR if it can decode and render as SDR. however, screen.video.hdrSupported would be false, because video would not be rendered as HDR (i.e. for video the UA sees the panel as SDR).

[chcunningham]

Why is the onus not on MC.decodingInfo() to report the HDR capabilities of the UA, which is the non-ultra cast device in this example. If the example were an ultra cast and any laptop w/o UA HDR support, MC.decodingInfo(), as opposed to hdrSupported boolean, would be used to discern HDR support of the laptop's UA.

Agree 100%. Sorry, I made a mess of the example again. The non-ultra cast (which is funcitoning as the UA) should signal lack of HDR support via decodingInfo(). What I meant to demonstrate is only that the Screen is HDR in both cases and could return hdrSupported = true in both cases. The MediaCapabilities / Screen pattern could then serve as a model for future graphics APIs (i.e. first check Screen.hdrSupported to see if its even worth asking more complicated questions, then proceed to ask about the hdr graphics rendering support via some TBD API just as you would video rendering with MediaCapabilities).

decoding.info might return true for HDR if it can decode and render as SDR. however, screen.video.hdrSupported would be false, because video would not be rendered as HDR (i.e. for video the UA sees the panel as SDR).

We bounced around on this a bit in the HDR centi-bug, but we ~settled on a wording that would actually cause decodingInfo() to return false in this case. The text part is here. In particular I'm looking at this bit...

The UA needs to be able to cause the output device to render the appropriate color, or something appropriately close enough. Valid inputs are defined by ColorGamut.

...and similar for eotf.

At least for video, I think this means we could describe the full matrix of UA + Screen support for HDR, but the hole for the graphics edge case remains (devices that support only HDR video).

I'm reading the stuff from @chrisn now (catching up on state of graphics world). Will make a follow up comment when I've digested that.

[gregwfreedman]

decodingInfo answers whether or not the UA can decode and render an HDR stream. screen.video.hdrSupported answer whether a site should use an HDR stream over an SDR stream.

if the UA can successfully decode an HDR stream and tone map it onto an SDR display then it should return true. if a site only has one stream and it's HDR, then it can use decodingInfo to know that the UA will do a reasonably good job presenting it.

if a site has both SDR and HDR streams (and decodingInfo returns true for both) it can use screen.video.hdrSupported to know which will provide the better user experience. if screen.video.hdrSupported is false, then the site would know to use the SDR stream since it was manually trimmed/graded for an SDR display. obviously, this would be better than using the HDR stream and letting the UA do the tone mapping to an SDR display.

The UA needs to be able to cause the output device to render the appropriate color, or something appropriately close enough. Valid inputs are defined by ColorGamut.

perhaps we should change this wording...

[gregwhitworth]

So to sum up, and please correct me if I'm wrong @gregwfreedman you're wanting decodingInfo can decode and render HDR content, even if that HDR content ultimately ends up being rendered in SDR; corrrect? In that case possibly adjust the wording to:

The UA needs to be able to cause the output device to render the appropriate color, or something appropriately close enough; possibly through down-sampling. Valid inputs are defined by ColorGamut.

We could also provide an example to show that a UA can convert from HDR content to SDR (eg: Rec2020 -> sRGB).

[chcunningham]

I'm reading the stuff from @chrisn now (catching up on state of graphics world). Will make a follow up comment when I've digested that.

I'm still reading... will post more soon (hoping suggests a direction on the core question of Screen.video vs simply Screen).

i think of screen.video.hdrSupported as not simply a property of the panel, but a property of the panel as seen by the UA

This resonates with me, and I think its consistent with the other things on Screen (e.g. our earlier discussion about Screen.width not being physical width, but width as seen by the UA). Do you think it motivates either: Screen.video.hdrSupported vs Screen.hdrSupported? To me it seems workable with either one.

decodingInfo answers whether or not the UA can decode and render an HDR stream. screen.video.hdrSupported answer whether a site should use an HDR stream over an SDR stream.

Similar to above, I think this interpretation is compatible with Screen.hdrSupported. Do you agree?

The UA needs to be able to cause the output device to render the appropriate color, or something appropriately close enough. Valid inputs are defined by ColorGamut.

perhaps we should change this wording...

; possibly through down-sampling.

We could also provide an example to show that a UA can convert from HDR content to SDR (eg: Rec2020 -> sRGB).

I support re-working wording. The original wording probably did not intend that decodingInfo(HDR) only returns true when an HDR display ("output device") is attached. I think the output device is only mentioned as an attempt to clarify that full support should be there if one were to be attached (i.e. the UA is doing its part prior to handing off to the Screen). As is, I think the wording runs too close to the former meaning.

Buuut, I think we should also be careful to clarify that, when and HDR display actually is attached, tone-mapping to SDR does not meet the bar for "supported=true".

Also, a thoughts on tone-mapping: you can meet the criteria above (HDR looks right on an HDR screen) while also not having tone-mapping. IIUC this is the situation with Chrome today. HDR vp9 or av1 will be accurate on HDR displays, but looks washed out on SDR displays. We may eventually implement tone-mapping but it hasn't been a priority because sites that send HDR so far have their own SDR grading they prefer to fall back on. I'm interested to hear more discussion on whether we think tone-mapping should be spec mandated in light of the above.

A little aside on this bit:

in the case of non-Ultra Chromecast, decoding.info might return true for HDR if it can decode and render as SDR. however, screen.video.hdrSupported would be false, because video would not be rendered as HDR (i.e. for video the UA sees the panel as SDR).

For this particular example the non-ultra chromecast isn't able to do any tone mapping. It doesn't understand the HDR eotfs, so any decodingInfo(HDR) call would return false. But, its also true UA doesn't view the Screen as HDR (for video or otherwise), the Screen property may as well return false too.

[gregwhitworth]

Buuut, I think we should also be careful to clarify that, when and HDR display actually is attached, tone-mapping to SDR does not meet the bar for "supported=true".

Before we try to adjust the text again, let me see if I can outline what I would expect to be returned with an HDR display and a SDR display.

	ColorGamut	HDR Display	SDR Display	Tone-map HDR to SDR
decodingInfo	rec2020	true	true	true
screen.video.hdrSupported	rec2020	true	false	false

Buuut, I think we should also be careful to clarify that, when and HDR display actually is attached, tone-mapping to SDR does not meet the bar for "supported=true".

Is this something that happens? EG: I have HDR content -> HDR supported UA -> HDR display and it gets tone-mapped to SDR?

If that is the case then somewhere I would expect this to actually return false because somewhere along the way something isn't technically supported.

[chcunningham]

Re: core question, Screen.video or simply Screen...

The Screen.video shape is entirely about solving two plane TV problem. I've been trying to consider this from all angles, reading up on the HDR graphics proposals and discussing offline with @jpiesing (HbbTV) and @lhalliwell (Cast).

My conclusions: 1) Surfacing the two planes on the web platform (Screen.video) is at odds with existing platform APIs. 2) Forcing TVs to use existing APIs (Screen, device pixel ratio, ...) may come at a cost. I think the cost is manageable and the workarounds are viable. 3) TV manufacturers are projected to continue building TVs this way, with graphics lagging video, as a cost / use-case-priorities calculation.

Let's get into it...

1) Surfacing the two planes on the web platform (Screen.video) is at odds with existing platform APIs.

Even before MC, having 2 planes breaks video's interaction with existing web apis like CSS and WebGL. Luke sums it up well here https://github.com/w3c/media-capabilities/issues/133#issuecomment-537691852.

This whole setup is obviously a bit yucky and not fully featured, i.e. there are things expressible on the web that are impossible to display this way (examples: rotating video at arbitrary angles, applying transparency to video with other content visible beneath, using a video as a WebGL texture).

Adding Screen.video doesn't address these breaks, but it does give TV makers cover to continue down this path. Conversely, Screen.hdrSupported is consistent with existing web APIs, which do not distinguish between "video" and "graphics" - its all part of the same Document and CSS object model. We can already find use cases for Screen.hdrSupported emerging from that graphics proposals. For ex, see here for a Canvas HDR use case.

2) Forcing TVs to use existing APIs (Screen, device pixel ratio, ...) may come at a cost. I think the cost is manageable and the workarounds are viable.

In both this issue (hdr) and #133 (height, width), forcing TVs to use existing APIs (Screen without .video, or device pixel ratio media queries) means they must lie about one of the planes. Ex: reporting Screen.hdrSupported=true or reporting a device-pixel-ratio of the output display (rather than graphics raster) would both be lying about graphics to benefit video.

I don't think this lie will be much practical harm for HDR images, CSS, or Canvas. Supporting any of these involves work at all levels of the stack: decoding, compositing, javascript interfaces etc.. Given the graphics plane limitations, TVs are unlikely to implement even basic support for the above. Its easy for apps to detect that a new HDR Canvas API is not implemented, or that a given image format is not supported (ex for webp here).

The cost of lying about device-pixel-ratio is a little less speculative. Web authors already use this mechanism to choose higher resolution images (retina displays, phones). Choosing a 4K image that's actually downsampled to a 1080p raster wastes bandwidth and @jpeising tells me this could even OOM a low end TV (more so for 8K). The likelihood of this will depend on web content. 4K images are pretty rare today. If TV apps start using them, perhaps it's fair to assume this will be preceded by (or precipitate) a change in TV manufacturing to add support.

I don't think the costs above outweigh the downsides of standardizing 2-planes in the web platform (see point 1). In worst-case scenarios, TV manufacturers can still resort to private APIs. This has its own fragmentation cost, but I think that aligns incentives correctly. Compared to a web standard working group, the people who make TV apps truly have the ear of TV manufacturers. If they resent the fragmentation of private APIs (or the breakage of video / graphics interaction in existing APIs), they are a powerful voice for change. This could mean coalescing around a common private API (a TV standard), or a push to have TVs put graphics and video in a unified plane. Which segues nicely into my next point...

3) TV manufacturers are projected to continue building TVs this way, with graphics lagging video, as a cost / use-case-priorities calculation.

@jpiesing tells me this isn't going away in a hurry. Video, not graphics, is the primary TV use case. TV manufacturers make choices accordingly, keeping costs low.

[chcunningham]

@gregwhitworth I think your table looks correct.

Is this something that happens? EG: I have HDR content -> HDR supported UA -> HDR display and it gets tone-mapped to SDR?

I agree, this would probably be a bug (not aware of any instance). On second thought I'm open to wording that doesn't explicitly mention this is a bug.

Feel free to iterate on the wording here without a heavy PR update.

[jpiesing]

It won't surprise anyone that I don't agree with @chcunningham about Screen.hdrSupported vs Screen.video.hdrSupported. Why should a new API force the major category of media consumption devices 1, 2 to lie about their device capabilities with a media related API?

It's correct to say this is a cost issue but it's a fairly extreme one. Increasing the graphics resolution from 1280x720 to 1920x1080, 4K or indeed 8K requires both extra RAM and a more powerful GPU if the same graphics performance is to be maintained. TV sets are a very price-competitive market and under continual cost pressure.

If a device with 4K HDR video but 2K SDR graphics has to pick one to return then I agree that it will return 4K HDR so that content apps can offer the correct media for the device. It will mean that, in a few years time, apps that want to do 4K images or HDR images won't be able to rely on what the APIs return. Do we really want to start building up this problem for the future?

[mwatson2]

+1 to @jpiesing's comment. The Web Platform should accommodate fundamental hardware limitations of the target hardware plaforms (unless we are saying that the web is only for desktop / laptop / mobile hardware class devices and all other devices should level up to that).

The issue also exists, though not in exactly the same way, for any HDR laptop today and will continue to exist until CSS adds HDR and all browsers supporting HDR video implement that.

There are other ways that TV browsers don't support all the capabilities of the web platform, some of which are software limitations which we could expect people to address and some of which are fundamental hardware limitations. We should address the latter.

[chcunningham]

IMO new hardware classes are always welcome on the web, but they shouldn't be able to redefine the DOM in ways that break existing APIs (add device pixel ratio to that list).

Again, the cost of maintaining video/graphics continuity seems low. If TV graphics being 1080p and SDR is a fundamental and unavoidable limitation, then TV html apps makers will never attempt to use 4K and/or HDR graphics. On the other hand, if TVs in the future support HDR and high resolution graphics, apps will of course use them, but this indicates the platform hw limitations have changed and we'll be better off not having monkey patched 2019's TV platform limitations into web apis.

The issue also exists, though not in exactly the same way, for any HDR laptop today and will continue to exist until CSS adds HDR and all browsers supporting HDR video implement that.

This is fundamentally different. There are open questions about what it will mean to composite HDR video alongside SDR graphics, and what HDR graphics APIs will ultimately look like. But none of those have motivated a separate "video" API on Screen, and HDR video on laptops is not fundamentally at odds with other APIs on the web platform. Screen.video is proposed as a solution only for the 2-plane problem on TVs.

[mwatson2]

@chcunningham The problem with todays HDR laptops is similar, because the CSS APIs indicate only SDR support (because there is no way defined for them to do otherwise), yet HDR video is supported. And those same devices can be in a configuration where HDR decode and rendering is supported but the screen is not HDR-capable (HDMI-connected SDR screen). On such a device / browser - which will be in the field for years to come - how are we to decide whether to deliver SDR or HDR video ?

Regarding TVs and other devices used in large part for TV viewing (set-top-boxes, HDMI sticks etc.) for a long time there will be a diversity of capability in the field. Many such devices have long lifetimes (7+ years) and devices with different video / graphics capabilities will continue to be shipped for many years to come. Your characterization of the choice between TVs having disparate video / graphics and having full HDR support is a false dichotomy - both will exist for a considerable time.

There will (eventually) be high-end TVs., STBs, streaming sticks with HDR graphics support and services like Netflix will want to make use of that capability as we do today on some games consoles. But the devices with separate video / graphics will remain in the field and continue to be shipped too.

Regarding:

IMO new hardware classes are always welcome on the web, but they shouldn't be able to redefine the DOM in ways that break existing APIs (add device pixel ratio to that list).

There is no need to "break" existing APIs, but there is a need to recognize that existing capability assumptions in the web platform could have been based on particular hardware classes and if you are serious about enabling use of the platform on other hardware classes you sometimes need to provide new capability detection.

Here, though, we are not saying that a capability previously always available will newly not be available in some cases and so now needs to be detected when previously it could be assume. We are saying that a new capability (HDR), not previously available in the web platform, is sometimes available only in a specific context (video). It is not a new thing for the platform that a capability that in principle has generic application (in this case to video and graphics) might in practice only be available in a specific context - this happens all the time.

[jpiesing]

IMO new hardware classes are always welcome on the web, but they shouldn't be able to redefine the DOM in ways that break existing APIs (add device pixel ratio to that list).

IMHO requiring implementations to claim to support 4K HDR graphics when they in fact only support 2K SDR graphics is breaking existing APIs far more than what we're talking about here. What we're talking about here could be achieved with one property!

Again, the cost of maintaining video/graphics continuity seems low. If TV graphics being 1080p and SDR is a fundamental and unavoidable limitation, then TV html apps makers will never attempt to use 4K and/or HDR graphics.

While there are certainly apps that are specific for TVs, there are also plenty of app creators who want a single HTML app to work as widely as possible including TVs.

On the other hand, if TVs in the future support HDR and high resolution graphics, apps will of course use them,

I'm sure graphics will get better in future TVs but I'm equally sure that video will get better as well (e.g. 8K). I would not expect graphics to catch video - in this type of device, media consumption is more important than graphics and that's not going to change.

but this indicates the platform hw limitations have changed and we'll be better off not having monkey patched 2019's TV platform limitations into web apis.

I'm uncomfortable with this characterisation of the problem when IMHO what we're really talking about here is one property.

[chcunningham]

I'm uncomfortable with this characterisation of the problem when IMHO what we're really talking about here is one property.

This overlooks most of the issues I've raised. The one Screen.video.hrdSupported property enshrines separate video/graphics planes which is not backward compatible with a number of existing APIs

• device pixel ratio (resolution media query)
• Screen.width and height
• Various styling of video not supported by a hole punch
• Video as a texture in WebGL / Canvas

IMHO requiring implementations to claim to support 4K HDR graphics when they in fact only support 2K SDR graphics is breaking existing APIs far more than what we're talking about here. What we're talking about here could be achieved with one property!

The reason this doesn't convince me is that no one is doing 4K or HDR graphics atm and when HDR graphics become available, we can anticipate idiomatic ways to feature detect their support (examples above). This leaves only the resolution issue. Respectfully, I don't think that sits heavier on the scale than the breaks I list above.

[chcunningham]

In reply to @mwatson2

... On such a device / browser - which will be in the field for years to come - how are we to decide whether to deliver SDR or HDR video ?

Screen.hdrSupported can tell you about the output device, which serves both video and graphics use cases in a future compatible manner. For video, sites still must call MediaCapabilities decodingInfo() determine if their format is supported decoding/rendering is supported. A similar pattern will exist for graphics. Images already has this mechanism, and I expect the draft APIs to follow suite. In the interim, it is easy for sites to detect that the HDR graphics APIs are simply unimplemented.

Here, though, we are not saying that a capability previously always available will newly not be available in some cases and so now needs to be detected when previously it could be assume. We are saying that a new capability (HDR), not previously available in the web platform, is sometimes available only in a specific context (video). It is not a new thing for the platform that a capability that in principle has generic application (in this case to video and graphics) might in practice only be available in a specific context - this happens all the time.

HDR is a new thing, yes. But this argument overlooks the breaks I listed above. For example, the combination of width / height and device pixel ratio is actively assumed to describe video just like any other element of the DOM. You are now saying that it needs to be detected by some other mechanism in a non backward compatible fashion. This is a "break".

[chcunningham]

There will (eventually) be high-end TVs., STBs, streaming sticks with HDR graphics support and services like Netflix will want to make use of that capability as we do today on some games consoles. But the devices with separate video / graphics will remain in the field and continue to be shipped too.

I mean to acknowledge that these devices will linger (this is point 3 in my longer post from yesterday). Future hardware gives me some comfort, but my argument does not hinge on there being a 100% unified video/graphics TV future. Please see my last paragraph in the reply to Jon above as to why I still don't find it a strong enough reason to pursue Screen.video.

[mwatson2]

Screen.hdrSupported can tell you about the output device, which serves both video and graphics use cases in a future compatible manner.

The problem with this is that CSS will just have the same problem in reverse:

• First, Screen.hdrSupported will deployed and used to detect that it's worth delivering HDR video to a device
• Later, CSS will define support for HDR colors. This will be detectable, of course, but this is more like our decodingInfo. There can still exist scenarios where tone mapping down to SDR is supported and the graphics plane is SDR but the video is HDR. Screen.hdrSupported will be true in this case for backwards compatibility with the point above. So people will be arguing for Screen.graphics.hdrSupported ...

The one Screen.video.hrdSupported property enshrines separate video/graphics planes which is not backward compatible with a number of existing APIs

• device pixel ratio (resolution media query)
• Screen.width and height
• Various styling of video not supported by a hole punch
• Video as a texture in WebGL / Canvas

Separate video / graphics support is not something we're introducing ('enshrines') - it exists as a fundamental property of an important class of devices. Unless you are saying that the W3C doesn't consider those devices as first class citizens in the web platform ecoystem ? But then why do we had the Web & TV Interest Group, CTA WAVE etc. and all the other things over the last 8 years that suggest otherwise ?

Some restrictions are easier to deal with than others. If I do have a site that relies on advanced styling / rotation / video-as-a-texture etc. then I am not surprised when that doesn't work optimally on a TV. I would not design a site that way if I expected a significant number of users on TV platforms. HDR vs SDR, HD vs 4K, by contrast, is just a matter of picking the right stream.

I agree that there could be use-cases for capability-detecting those other things, but I don't see anyone arguing for that kind of capability detection, so I think we can assume those use-cases are theoretical only. We could reasonably ask CTA WAVE about that.

Regarding resolution (or more specifically devicePixelRatio, since width and height in CSS pixels don't change based on resolution - CSS pixels have a physical size): I think we either need video.width and video.height in device pixels as originally proposed or you could have video.devicePixelRatio. Requiring the existing devicePixelRatio to reflect the higher video pixel density will result in sites pulling down high resolution images unnecessarily.

[jernoble]

@jpiesing said:

I'm uncomfortable with this characterisation of the problem when IMHO what we're really talking about here is one property.

Well, this is what I believe is the heart of the issue: this is not about just one property. Even if this narrow property was defined in terms of the "video plane" (for devices with separate video and graphics plane), pages would still be unable to make a functional media player without out-of-band information, due to all the other inconsistencies between the two planes. So that would require another property, and another, ad nauseam, until we end up reproducing much of the CSS OM in terms of the "video plane".

So, here's one proposed resolution to this issue:

1. This spec should make a normative reference to the CSS Media Queries specification.
2. This property should be moved from a DOM property on window.screen to a "media feature".

This has three benefits:

• It automatically gives pages the ability to detect when "hdrSupported" changes in script, without having to specify an addition of events to window.screen.
• It allows some pages to add HDR support in markup, rather than requiring all pages to drop to javascript.
• "Media features" are scoped to "media types", which leads to item 3 in this proposed resolution.

3. TV manufacturers which have separate characteristics for their "graphics plane" and "video plane" should signal such through a separate "media type".

This could be an existing, but unspecified, "media type" such as "tv", or a new "media type" could be specified. This media type would scope all feature queries, including ones for display resolution, color gamut, and this new "hdrSupported" feature. Pages could then query for, e.g., @tv { (hdrSupported) and (resolution >= 2dppx) } in addition to @screen {...} to get an accurate understanding about the underlying characteristics of the system.

This third step should happen in the CSS WG. In talking with @hober offline, the best way to get traction inside the CSS WG would be to join the CSS WG. Are the relevant TV UA representatives members of the CSS WG?

[jernoble]

@mwatson2 said:

I think we either need video.width and video.height in device pixels as originally proposed or you could have video.devicePixelRatio. Requiring the existing devicePixelRatio to reflect the higher video pixel density will result in sites pulling down high resolution images unnecessarily.

No, we should not do either. Those APIs return values in CSS pixels; if devices are giving incorrect definitions of what a CSS pixel means, that should be fixed in CSS, not in the Media WG.

[mwatson2]

No, we should not do either. Those APIs return values in CSS pixels;

Screen.video.width and Screen.video.height were proposed to be in device pixels. Yes, Screen.width and Screen.height are in CSS pixels so the difference could be confusing, but there's nothing else technically wrong with it.

Nevertheless, Screen.video.devicePixelRatio would also work.

[jernoble]

Nevertheless, Screen.video.devicePixelRatio would also work.

As would @tv { resolution >= 2dppx }and would solve many more problems at once.

[jpiesing]

Nevertheless, Screen.video.devicePixelRatio would also work.

That was what was in my mind as a solution.

[vi-dot-cpp]

Thanks everyone for your input on this. Would you all be opposed to my breaking Screen changes out of PR #124 for the sake of modularity? It may help to present MC (decoding & rendering) and Screen (display) information in a more digestible way for everyone.

[chcunningham]

In reply to @mwatson2

Separate video / graphics support is not something we're introducing ('enshrines') - it exists as a fundamental property of an important class of devices. Unless you are saying that the W3C doesn't consider those devices as first class citizens in the web platform ecoystem ?

The debate is how/if to introduce this notion into the web platform. I have acknowledged the realities of hw on these devices and their weight in the media consumption world. None of my arguments claim that TV is an unimportant category.

In reply to @jernoble, I think the media type solution is really intriguing! I have some homework to do re: media-types, but at first glance strikes me as an elegant solution.

[chcunningham]

Thanks everyone for your input on this. Would you all be opposed to my breaking Screen changes out of PR #124 for the sake of modularity?

Fully support. The PR does remove existing (very dated) references to Screen (luminance, etc). The PR could continue to drop those and simply not add any new hdr field.

[mwatson2]

As would @tv { resolution >= 2dppx }and would solve many more problems at once.

Anything that works is good for me!

I do think that this group should own finding a solution, unless we have a clear signal that another group will get it done on our timescales. Of course we should ask CSS to weigh in, but we should not be dependent on them to move forwards.

[vi-dot-cpp]

@chcunningham Fully support. The PR does remove existing (very dated) references to Screen (luminance, etc). The PR could continue to drop those and simply not add any new hdr field.

PR #124 has been rid of references to HDR for Screen. I have also clarified the definition of ColorGamut in VideoConfiguration. Are there any other blocking issues on this PR (given that it is now constrained to changes to MC)?

Regarding the two-plane problem on Screen, please allow me some time to do some research. Thanks again to everyone for chiming in.

[chcunningham]

PR #124 has been rid of references to HDR for Screen. I have also clarified the definition of ColorGamut in VideoConfiguration. Are there any other blocking issues on this PR (given that it is now constrained to changes to MC)?

Hopefully not :) - I'm taking a look right now. Thanks a ton.

[tidoust]

@jernoble said:

This could be an existing, but unspecified, "media type" such as "tv", or a new "media type" could be specified.

The Media Queries specs is clear that media types are now seen as the wrong level of abstraction to discriminate between devices: "Unfortunately, media types have proven insufficient as a way of discriminating between devices with different styling needs".

In particular, the tv media type is marked as deprecated and the section ends with the note: "It is expected that all of the media types will also be deprecated in time, as appropriate media features are defined which capture their important differences."

[chcunningham]

here's the relevant comment for TV

Others, such as tty or tv, expose useful differences from the norm of a full-featured computer monitor, and so are potentially useful to target with different styling, but the definition of media types as mutually exclusive makes it difficult to use them in a reasonable manner; instead, their exclusive aspects are better expressed as media features such as grid or scan.

I'm not sure I understand it. what is it about mutual exclusivity that makes it hard to use in this case?

[gregwhitworth]

Ok, I've now had a chance to read through the flurry of comments since yesterday's telecon. And I was hoping to quickly summarize what I believe to be the base problem that we as a group need to resolve on which direction to go. At that point, I am happy to take these same two options to the CSSWG on a telecon and outline the issue and get their feedback.

1. Only have a single hdrSupported bool (whether that's a CSS Media query or a property of screen)

   • Pros: This keeps the API shape clean and more manageable for implementors & web devs
   • Cons: Requires TVs to lie about every API that has to do with the graphics plane in televisions today

2. Have two bools for a graphics vs video plane (whether it's a CSS MQ or a property of screen & screen.video)

   • Pros:
     • Recognizes the realities of TVs as they exist today
     • Even if TV graphics plane hardware catches up to laptops/consoles/desktops/mobile devices owners don't replace TV's at the same interval as the other device category. As of 2014 it ~4-6 years
   • Cons:
     • Requires duplication of DOM APIs necessary to provide a solid solution to TVs for Applications (graphics plane) and high quality media content (video plane). [I think it's worth itemizing ALL of these]
     • Creates in-congruent API shape

Does everyone agree that those are our two potential approaches? If that is the case I think we as a Media WG first need to get to an agreement regarding which path we think is the best path forward (even if it's 70/30). I can then take this summation to the CSSWG, I'd prefer to let them know where we are with a solid summation rather than dropping a 40 comment thread on them.

Please denote if this summation is missing anything and with which option you prefer, if you're undecided - state as such. After a week I think we should complete the summary and then do a call for consensus on the WG mailing list.

@jernoble @mwatson2 @chcunningham @tidoust @jpiesing

[mwatson2]

Hi @gregwhitworth, I agree these are options for HDR, but I don't fully agree on or fully understand the pros and cons. Also resolution needs to be part of the discussion.

Option (1) does not just require TVs to lie but also any browser supporting HDR video but not HDR graphics, which would be any browser supporting HDR video today and until CSS standardizes support for HDR graphics and it is universally implemented.

Another con of option (1) is that is prejudges that CSS support for HDR graphics will be indicated by this same boolean vs CSS having the same argument as us in reverse.

Option (2), as @jpiesing points out, it is unlikely that TV graphics plane hardware catches up with video, because video may move too. I'd note that the 4-6 years is an average and that for a major service to drop support for a device we need to wait for the 99th percentile or similar.

I do not understand what you mean by "duplication of DOM APIs". Could you elaborate ?

[chcunningham]

Thanks @gregwhitworth !

I would break out the media-query version of hdrSupported from @jernoble 's suggestion of a new media-type (or reviving the tv type). That is, Screen.hdrSupported and window.matchMedia( "(hdr: supported)" ).matches have equivalent pros and cons listed in 1. But @jernoble's proposal to surface tv video plane limitations in a separate media-type (e.g. window.matchMedia( "tv and (hdr: enabled)" ).matches would IIUC marry the pros of 1 and 2 with the cons of neither. The major con for is only that media-types are being deprecated (what luck). Still, I'm not ready to declare it DOA until they weigh in. It may at least be the least of 3 evils ;).

Option (1) does not just require TVs to lie but also any browser supporting HDR video but not HDR graphics, which would be any browser supporting HDR video today and until CSS standardizes support for HDR graphics and it is universally implemented.

@mwatson2 and I disagree on this point. Screen.hdrSupported does not imply support for graphics APIs that are yet to be implemented. When non-TVs implement new HDR graphics APIs, Screen.hdrSupported is ready to signal that HDR colors are worth using.

Another con of option (1) is that is prejudges that CSS support for HDR graphics will be indicated by this same boolean vs CSS having the same argument as us in reverse.

I agree this is a possibility, but I'm on the fence about whether its a con (feel free to list it). When we say hdrSupported = true, the separate video plane on TVs only makes this a lie wrt to graphics. Perhaps CSS is the right forum to resolve a disagreement about graphics.

I do not understand what you mean by "duplication of DOM APIs". Could you elaborate ?

I think he's referring to my / jernoble's comments about the other things that are broken by two planes. This list at least includes: Screen width/height, devicePixelRatio, possibly hdrSupported, and relevant media queries (e.g. resolution).

[gregwhitworth]

But @jernoble's proposal to surface tv video plane limitations in a separate media-type (e.g. window.matchMedia( "tv and (hdr: enabled)" ).matches would IIUC marry the pros of 1 and 2 with the cons of neither.

It does keep the bifurcation (eg: TV is standalone here) but I do agree that it is the best path forward out of the options.

Option (1) does not just require TVs to lie but also any browser supporting HDR video but not HDR graphics, which would be any browser supporting HDR video today and until CSS standardizes support for HDR graphics and it is universally implemented.

@mwatson2 and I disagree on this point. Screen.hdrSupported does not imply support for graphics APIs that are yet to be implemented. When non-TVs implement new HDR graphics APIs, Screen.hdrSupported is ready to signal that HDR colors are worth using.

I agree with @chcunningham here and my reasoning is that the primary scenario in the near future with regard to HDR graphics will be that of images, which we already have solutions for selecting appropriate assets in both HTML via <source> with the type, sizes attributes, and image-set in CSS.

If it is CSS color itself, then my expectation is that we'll introduce a new color function which by the design of CSS will get thrown away if it isn't supported; which would only get implemented if the UA can itself render HDR. That said, is this a common need? The primary area I can see value specifically with regards to CSS color would be that of gradients to reduce banding, but there may be others and am genuinely curious to hear them. I'm more than happy to help make progress on that if there is a compelling enough usecase and demand from web developers for it.

@mwatson2 are there any other graphical HDR scenarios that I'm missing as I'm happy for us to explore them to ensure that all are covered by the MQ proposed by @jernoble

[tidoust]

@chcunningham said

The major con for is only that media-types are being deprecated (what luck). Still, I'm not ready to declare it DOA until they weigh in. It may at least be the least of 3 evils ;).

Regarding media types, it's certainly fine to check with the CSS WG. Let me try to clarify some of the reasons for the deprecation because this has been discussed years ago already for handheld when smartphones started to appear.

A media type encapsulates a set of properties that appears common to target devices at a given set of time, some of them being implicitly defined. For handheld, this was something like small screen, limited bandwidth, low resolution, no support for a number of CSS features such as animations, etc. Typically, authors happily created trimmed down stylesheets for handheld devices that created good user experiences for small screens, and full featured screen stylesheets for larger screens.

Inevitably, a new product arises that has some of the same properties/constraints, but also new features that go beyond the understood definition of the media type, for instance better resolution (even though the screen may be only marginally larger) or much better support for CSS technologies. If that new device continues to apply the previous media type, new features won't be used and user experience will be sub-optimal, e.g. because very low-resolution images will be used. If the new device switches to the broader screen media type, new features get used, but that breaks some of the author's (not necessarily correct) assumptions (e.g. that screen is a large screen).

In the handheld case, the new device was the iPhone (and all smartphones since then), who decided to claim that they were screen devices, and not handheld devices, because they could handle the full-featured versions of the stylesheets and higher resolution images. That both enabled better experience down the road and broke content that authors had carefully crafted for phones using handheld initially.

In the end, the problem is that media types are too coarsely defined and not stable enough over time because they embed multiple dimensions (some of them being implicit assumptions). That essentially explains the move to media features, which allow authors to be explicit about the dimensions they target, and craft more responsive layouts.

I think the same "devices evolve over time" applies to TV by the way. For instance, several years ago, the CSS 2.1 spec described the tv media type as "low resolution". And yet we are today talking about 4K/HDR support, which seems more advanced in TVs than in other types of devices.

For an example of statement that cautioned about the use of media types during the "transition period" to media queries, you can have a look at the Use client-side detection capability detection where necessary best practice in the Mobile Web Application Best Practices recommendation, published in 2010:

"CSS Media Types allow different stylesheets to be associated with different media types (e.g. print, screen, mobile) and are traditionally used to repurpose content for these destinations. Since the list of recognized media types is limited, however, and devices are notoriously idiosyncratic in their interpretation of types, it is in general not a helpful technology in this context."

[chcunningham]

Really insightful! To make sure I follow, here's my guess at what a media-feature solution would have looked like for the original pre-iphone era animation challenges.

@media screen and (animations: 0) { 
... // different styling where animations not possible
}

Sadly, this seems like a simpler class of problem than the 2-plane situation. In our case, the feature-based solutions I see simply map to the Screen{.video} options (with their same pros and cons). That is, we can either not mention video (feature: hdrSupported = Screen.hdrSupported) or we can mention it (feature: hdrVideoSupported = Screen.video.hdrSupported). Same goes for the devicePixelRatio. LMK if there's a silver bullet lurking in the MQ syntax.

[tidoust]

LMK if there's a silver bullet lurking in the MQ syntax.

I don't see any for now.

Also, if I understand @jernoble's proposal in https://github.com/w3c/media-capabilities/issues/135#issuecomment-543280597 correctly, I'm not sure that reintroducing a tv media type would actually work as suggested. In particular, I don't think that "Media features are scoped to media types". At least until now, media features have been defined regardless of media types.

For instance, the definition of resolution is the same for all media types. In @media tv and (resolution >= 2dppx), the term resolution would still describe the "resolution of the output device", not the "resolution of the output device for video". Otherwise, @media all and (resolution >= 2ddpx) would have a different meaning depending on the actual device, which seems wrong.

[chcunningham]

In particular, I don't think that "Media features are scoped to media types". At least until now, media features have been defined regardless of media types.

I think @jernoble meant the same features would be defined in all scopes, but the values for those features would differ by scope. Ex: resolution in the scope of @media tv would query the dpi of the video plane, while resolution for @media screen would query the dpi of the graphics plane.

[chcunningham]

Discussed a little more at FOMs. @mwatson2 @jernoble @hober (and many others) present. We all agree the media-type solution is very appealing. @hober, also part of the CSS working group, didn't see the past deprecation as necessarily at odds with this new usage.

For the TV video plane, we discussed using @media video instead of tv, which has it has an existing (deprecated) meaning that doesn't match what we're now hoping to describe.

Going back to an earlier comment...

but the definition of media types as mutually exclusive makes it difficult to use them in a reasonable manner

Our proposal would require that video and screen types not be exclusive. TV's with two planes would need to match both 'screen' and 'video' media types for sites to query the features* of the graphics and video planes respectively. I'm hopeful that the different semantics here again suggest that we shouldn't dismiss the idea along the lines of the planned deprecation.

*I think the following media features could be expected to return different values under the video type on a 2-plane TV: color, color-gamut, color-index, device-aspect-ratio, resolution, height, width

[gregwhitworth]

The more that I've thought about this the more I'm curious if we need a new way to query the graphics plane at all. As I outlined above, every graphic scenario that I can think of already has a way in which to handle it. So the only thing we need are queries specific to video for HDR & Resolution.

I noticed that the <source> element within <video> can't harness the srcset nor the sizes attributes (although I didn't test this). This would give you the bifurcation you're wanting in a declarative manner which can be set on the media element via script. This would then be querying the video plane within TVs for

So through mime, srcset & sizes also being supported for <video> and bifurcating in the code where necessary (eg: TVs would only query the video plane for the <video> element) you should be able to accomplish any query you want. Does anyone see any holes in this? @mwatson2 @jernoble @hober @chcunningham @vi-dot-cpp

[mwatson2]

@gregwhitworth I’m not sure I follow the proposal. Could you provide example code that would decide whether to fetch the SDR or HDR version of the content - assuming media capabilities indicates support for both ?