Device Posture API [formerly Screen Fold API]

[darktears]

HIQaH! QaH! TAG!

I'm requesting a TAG review of the Screen Fold API.

New types of mobile devices (phone and laptop segments) are coming to market with the ability to fold the screen in some capacity, either by using foldable displays or multiple internal displays (MIDs). Examples include the Samsung Galaxy Fold, Motorola Razr, Lenovo ThinkPad X1 Fold. 

While the technology enabling these devices is rather new, there is a resurfacing of a trend started by ultrabooks and 2 in 1’s which have had different operating “modes” or postures depending on the hinge angle. To note a few, “book”, “tent” and “tablet” modes were available on some devices. All these postures are triggered depending on the hinge/fold angle. The above specification would expose the fold angle as well as the postures.

From enhancing the usability of a website by avoiding the area of a fold, to enabling innovative use cases for the web, knowing the fold angle can help developers tailor their content to different devices. It can also enable to detect different postures the device might be in.

• Explainer¹ (minimally containing user needs and example code): this spec + the domain
• Security and Privacy self-review²: here
• GitHub repo (if you prefer feedback filed there): here
• Primary contacts (and their relationship to the specification):
  • Diego González-Zúñiga Microsoft
  • Kenneth Rohde Christiansen Intel
  • Alexis Menard Intel
  • Laura Morinigo Samsung
  • Anssi Kostiainen Intel, Chair of DAS WG
• Organization/project driving the design: Intel and Samsung
• External status/issue trackers for this feature (publicly visible, e.g. Chrome Status): here

Further details:

• [X] I have reviewed the TAG's API Design Principles
• The group where standardization of this work is intended to be done ("unknown" if not known): Devices and Sensors Working Group -> https://www.w3.org/das/
• Existing major pieces of multi-stakeholder review or discussion of this design: Unknown
• Major unresolved issues with or opposition to this design: None at the moment
• This work is being funded by: Intel and Samsung

You should also know that...

Please also refer to https://webscreens.github.io/form-factors/ explainer for terminology and explanation of the problem space.

We'd prefer the TAG provide feedback as (please delete all but the desired option):

🐛 open issues in our GitHub repo for each point of feedback

[LeaVerou]

My initial thoughts:

Foldable devices have only just started emerging, yet this proposal is very much built around their current constraints (one fold, along a central axis), which makes me skeptical about how well it will stand the test of time. What if in the future we have devices that can fold in multiple places and axes? What if we can even have arbitrary folds? What if they can bend but not necessarily fold? I'm not sufficiently knowledgeable in hardware technology to know if this might ever be possible, but thought I'd flag it for those who are.

A few more thoughts regarding syntax:

• I think an angle media query is way too generic. Is it fold angle, screen tilt angle, or something else? I'd suggest going for fold-angle.
• If my understanding is correct, screen-fold-posture can be addressed via a combination of min-fold-angle and max-fold-angle. If so, perhaps it would be best for the higher level ranges to be added later, once we can observe what ranges authors actually use for these things?
• Does the environment variable need screen-*? What other fold could it be targeting if it were just fold-angle? (not a rhetorical question)

[darktears]

Thanks @LeaVerou for you feeback.

I believe that we will see devices with non-symmetrical folds, or more than one-fold. Though it's unlikely we'll see more than 2-3 folds due to the complexity of the hinge mechanism to be put into such a device. The mechanism takes space and at some point, one must value the space in the device vs battery life vs benefits of multiple folds.

• angle vs fold-angle, I don't have a strong opinion on it. It's just inside the namespace screen.fold so angle name seems obvious that it applies to the fold itself, or in other words the API call would be: screen.fold.angle.
• I don't think we want to encourage author to compute the posture based on the angle or angles. This would yield to inconsistent behaviors between sites because authors would decide differently when the device is used a laptop/unfolded/etc. Also, the posture determination is a lot more complex than just the fold angle, for example if the keyboard is magnetically attached or not, if the device is standing on a table (tablet vs used as a monitor). It's important that we let the OS determine and enforce the posture because it will do a better job than web authors. Lastly user would expect the device to react certain ways when they use the device, and they will be puzzled if a given website do not behave as the rest of the ecosystem.

[anssiko]

@LeaVerou thanks for your TAG review feedback. We've put this issue on the Second Screen WG/CG - 2021 Q1 virtual meeting agenda (23-24 Feb 2021) and any further feedback from the TAG received by that meeting will get discussed in there. You're welcome to attend.

I'd also like to share we're planning to publish a new Working Draft within the next two months and aim to get the TAG review feedback addressed in that version if possible.

[plinss]

I recall seeing other proposals from Microsoft that included further parameters like the gap between screens, has this work coordinated with that work?

I share Lea's concerns about multiple folds and multiple fold directions. I'd also like to see this work in the general case, and not be so specific to today's current folding devices. Furthermore, this should also be able to represent multiple monitors in the desktop case. For example, I have a main monitor and two side monitors at 30deg angles, why wouldn't that setup be represented in the same API as a folding laptop? (OS support for getting that data notwithstanding, the browser can always add it's own UI for screen configuration info)

As to the "not more than 2-3 folds" argument, I present: https://gizmodo.com/its-no-surprise-this-absolutely-obscene-7-screen-laptop-1846222859

My other question is why is this a separate ScreenFold interface, rather than additional properties of the existing Screen API?

[darktears]

@plinss both this proposal and the one from Microsoft are complimentary. The screen spanning proposal of Microsoft could be seen as additional hooks that are specific to dual screen devices. Dual screen devices do have an hinge and they do have postures as well.

The latest discussions within the working group is actually to streamline this proposal and expose only the posture of the device and thus removing the angle which so far has shown some limited use cases from the community (not to mention privacy concerns). I think this would make the future proof part of the comment

And yes I'm very well aware of the crazy 7 screen laptop but my comment was about the number of folds on the screen itself.

[plinss]

My point in bringing up the 7 screen laptop the that there's a gray area between a single folding display, and multiple displays that can be reoriented relative to each other. A device with two screens and a narrow hinge between them isn't much different from a single screen that folds. I don't see the benefit in treating those differently. There are also existing use cases of desktop environments with more than two screens, it'd be a shame to invent something new here targeted to newer devices that leaves all the legacy devices unconsidered.

[plinss]

Another aspect here is the relationship between the viewport and screen, on non-phone devices these don't align unless running in full screen mode, which is generally not the norm. What web authors really need to know is when and where the hinge is intersecting the viewport, e.g. if there's a single browser window on one side of a folded screen, the viewport isn't folded at all, why would a web page care about the screen fold? If the window partially spans a fold, then where the fold intersects the viewport is what's important. Also consider the case of a browser window being dragged across a folded laptop screen, then position of the fold will be changing dynamically, the web content should be able to adapt to that.

What would be more valuable to web authors is an API that accounts for how the viewport is behaving than the screen. An API describing the physical screens is useful in some limited cases, but an API describing the viewport's condition and behavior would be more valuable when adapting web content to the available display.

[atanassov]

@darktears can I ask that you respond to @plinss comments so we can continue helping you with this review?

[darktears]

@atanassov @plinss the API has significantly been reshaped with now just exposing the general posture of the device. Most of the comments in here are now void. Regardless if the device has multiple folds or one, there will be only one single posture defined at any time. The posture is decided by the OS using various inputs (such as fold angle values, orientation, sensors, keyboard etc). With that new scope in mind, I believe the review will be simpler, I encourage the people in this thread to have a look at it.

[atanassov]

Thanks for the updated explainer @darktears. The simplified proposal is very straight forward and much better starting point as it allows lots of future extensibility. Couple of points that came up during our discussion.

1. (more of a nit) The icon of the explainer (of feature?) looks like an 'f' made out of multiple folds, yet the proposal doesn't offer such capability/posture. Besides this nit, once multiple folds are available, have you thought about how would such postures be described?
2. The are no examples that demonstrate how the feature is used or useful to authors. Without these motivational examples the current proposal comes across lacking value. Let us know when you've addressed these so we can do another round of reviews.

[darktears]

Thanks for the comments @atanassov.

1. We can update the logo if needed but at this moment we don't mention multiple folds because there isn't any device out-there not because we don't support it. Regardless of the number of folds I think the device will always have a single posture, but I don't think we have given thoughts about the names if a new posture would be invented. Sometimes having multiple folds will not influence the outcome of the posture, sometimes it will it really depend on the how the layout of the folds will be. If the device is a three panels postcard alike, then I don't think it would be different in terms of posture. But it could be that the third area/segment is folded and used as an auxiliary/addon screen and thus we could come up with a new posture like tablet-with-secondary-segment or whatever. We can always extend the enum.
2. We have https://w3c.github.io/device-posture/#examples but I agree it's a bit limited. I'm sure we can beef up the examples.

[plinss]

One of the things we're looking to understand (via the examples) is how the author is expected to respond to the various postures. I'm concerned that the current approach doesn't give enough information to alter the page layout in ways that are useful.

That concern could be addressed by stronger definitions of the posture, e.g. does 'book' always mean the screen is folded vertically (relative to the devices current posture, even though the fold could be horizontal or vertical in nature), and the fold is in the center? Is 'tent' always a horizontal fold in the middle? etc.

[darktears]

We have updated the specifications significantly and now we only expose 3 posture types. Folded, Continuous and Folded-Over. This will remove the complexity while allowing the original intent.

[cynthia]

Thanks for the update. We've discussed this to some extent, and the changed design with limited posture types looks good a few of us were confused about the utility of this proposal. I think there is a bigger picture in play here, but we lack the context by looking at a small piece. It would be really helpful to us (and the future users of this standard) to have some sort of landscape explainer on how each little piece fits together to make this appealing to the end-users. Would it be possible to provide such context? Thanks.

[darktears]

Thanks for the feedback.

We did provide an example in the spec and we have various demos online.

Say you want to implement something like this?

How can you detect when to do the split mode? The device doesn't change its orientation, only the hinge moves.

The API will tell you that you're now folded and that you can enable the Split UX.

The same would apply if the device fold over and you would want to turn off the screen on the non-user-visible side or for example if you wanted to do a two-player mode. This is what people call "tent mode".

Now in the future you may have multiple folds and yes we need a way to know where they are and that's what the Window Segment API is about. We have a better detailed explainer over here: https://webscreens.github.io/form-factors/

At this point native is successfully leveraging the posture definition as we laid them out here. Android has been shipping a simple posture API which is being used by several native apps like Google Duo, YouTube, Samsung native apps and so forth.

[atanassov]

Thanks for the quick responce @darktears. Our main concern is mostly from the author point of view. It is not clear how exposing posture media feature makes it possible for authors to adapt their UX to it. Currently they need to (also seen in your examples) query the spanning features and env variables in order to adjust the document layout.

[atanassov]

To your question about How can you detect when to do the split mode? The device doesn't change its orientation, only the hinge moves. Isn't this equivalent to detecting the spanning change and direction?

[darktears]

Both are complimentary specs. They should be used together.

On a dual screen device when spanning the hinge may not move or the device is flat, yet you need to avoid hinge occlusion, Window Segment/CSS Spanning is there to help you. In this particular device how would you differentiate the flat case where you have two window segments and the fold state where you have two window segments as well? The author may choose to provide a different UX (for e.g half folded could be used to display game control on the bottom section like a gaming handheld which isn’t applicable if the device is flat).

[atanassov]

@darktears thank you for the quick turnaround and clarification. This was the exact use case we were missing and letting us question the overall value.

Further, the examples in your current spec are showing non-hinged devices and that adds to the confusion. I would argue that for such devices I don't need this API since the posture change is detectable through spanning going from 1 (flat) to 2 (folded).

One last concern that we hope you will continue to work on as you make progress is support for multi-hinged devices. The explosion of postures is close to the number of permutations between screens and hinges, something that adds up very quickly.

At the end we are happy to close this review as satisfied. Thank you for working with us, taking and addressing all of the feedback provided and most of all being patient through the process. We look forward to seeing further progress with this work. Good luck.