Presentation Testing API

[markafoltz]

Currently a good portion of our web platform tests require manual intervention. This task tracks the work to implement a testing API to allow the tests to:

1. Create and remove a fake presentation screen
2. Select or cancel presentation
3. Start presentation from the browser
4. Implement the "stash" functionality?

We should consider prior work by WebVR and other device APIs here around testing APIs.

[anssiko]

Looping in @foolip with whom I've discussed the challenges of manual w-p-t testing of device APIs, particularly Vibration API and Battery Status API.

@Honry has been championing the said test suites recently, and may also have some ideas for improvement to the manual testing of Presentation API.

This might be another good concrete case to discuss in the web-platform-tests breakout session at TPAC on Wednesday. Or perhaps manual testing challenges would warrant a separate breakout session.

[tomoyukilabs]

4. Implement the "stash" functionality?

+1. The reasons are:

• Both the controller and the receiver use pre-defined constant UUIDs, which might cause conflict between simultaneous access to stash.
• Stash requires HTTP polling mechanism for asynchronous communication, which might cause impact on WPT server's CPU load and network traffic.

[louaybassbouss]

@mfoltzgoogle just one question if we allow to create a fake presentation for testing purposes it is still the same underlying implementation of the API in Browser? we need to make sure that the behaviour of the API when presenting on real display is the same as when presenting on a fake display.

[markafoltz]

@louaybassbouss The Presentation API implementation in Chrome consists of multiple layers spread across different process types. Chrome runs Web Platform Tests on the content/ layer, which (in Chrome) excludes any device-specific logic. The "fake" screen would have to be implemented in content/; so parts of the implementation of the Presentation API would be shared between "real" screens and "fake" screens. Others would not.

As part of supporting "fake" screens, we may decide to move more of the implementation into content/ to avoid having to duplicate the logic between "real" and "fake" screens. This would improve test coverage for those aspects of the implementation.

I can provide more concrete feedback once we have a proposal for the functionality of a "fake" screen.

[markafoltz]

I took a quick look at the status of similar APIs that require some device mocking behavior.

• WebUSB
  • WebUSB Testing API
  • Allows creation of fake devices and choice of a specific one to simulate user permission
• WebVR
  • Microsoft proposed a headset simulation API, but it appears to have never landed upstream; and the WPT suite appears to be starting over from scratch for WebVR 2.0.

• Web Bluetooth

  • Testing API Design Doc (partially implemented)

  This thread from blink-dev is also relevant to the status of these tests from May of this year. There was some more discussion and a follow-up call in earlier this year on the Test Infrastructure list, specifically about the WebUSB proposal.

[markafoltz]

I think the logical next step here is to define the requirements and IDL for a testing API, similar to WebUSB. The initial scope I will look at is supporting cross-browser automation of Web Platform Tests. I'll just post it to this issue for discussion (not starting a formal document or anything.)

[markafoltz]

@foolip @Honry

Here is a draft of what I am thinking to start discussion. I think they will allow conversion of the manual Presentation API tests to automated, but I may have missed something. PTAL.

https://github.com/mfoltzgoogle/presentation-test-api/blob/master/presentation_test_api.md

I am not committed to any particular implementation; there has been some discussion of implementing testing APIs on top of WebDriver, versus bindings to native implementations.

[markafoltz]

As we're talking about a new API (or WebDriver) surface, I'm marking this as a v2 feature. It's also a higher priority work item for us.

[foolip]

@mfoltzgoogle, as of yesterday http://web-platform-tests.org/writing-tests/testdriver.html is an option as well. That requires first defining a WebDriver API, and then wrapping it in something that's ergonomic when writing tests.

The upside is that web developers will also get the capabilities. That being said, a JS API that's enabled by WebDriver is also an option, if the API is too complex to represent as a WebDriver API. @JKereliuk might be able to give further advise.

[markafoltz]

From https://www.w3.org/2017/11/06-webscreens-minutes.html#x13:

• ACTION: @mfoltzgoogle to refine the Presentation API Testing API and come up with a WebDriver compatible spec
• ACTION: @mfoltzgoogle to file issue about whether "fake" screen is a "real" screen or can be a mock screen that responds to messages