Remote Playback API tests and implementation report

Honry commented 7 years ago

Import comment(https://github.com/w3c/remote-playback/issues/87#issuecomment-308479127) from @mounirlamouri:

The issue with this API is that you will have to use a Chrome Cast compatible device to test the API on Chrome but another browser might not support Chrome Cast. The test preconditions will have to start with something a bit hand-wavy like "Have a device compatible with the browser on your WiFi". Because only Chrome (maybe other Chrome-based browsers) has an implementation for this at the moment, it would be hard to be clearer.

Honry commented 7 years ago

@louaybassbouss, I know you did really good on test automation for Presentation API, the problem we encountered in Remote Playback should be similar, could you please share your experience for us?

Honry commented 7 years ago

@louaybassbouss, friendly ping.

louaybassbouss commented 7 years ago

@Honry sorry I missed your request from 21 Jun thanks for the reminder. Yes the setup for testing the Presentation API and RemotePlayback API Tests on Chromecast should be similar. I would say that the tests of RemotePlayback API are easier to implement since you will not deal with different Presentation URLs as in the Presentation API (You can have a look in [1] where two different Presentation URLs are specified, one for cast receivers and one for other receivers that can render a web page). In RemotePlayback API you don't need to pass any URL which makes a lot of things easier. To test on Chromecast (or any other cast receiver like Android TV) you need just to make sure that your PC and the Receiver devices are in the same Network. But you need to keep in mind when you are writing the tests to not use any code that is specific for chromecast or any specific technology, but only according the API specification. To create a test report of RemotePlayback API e.g. for Chrome Browser, the tester (for example the person who is responsible to create the report) must have the information about which receiver devices are supported by the Browser under test. The tester musst also ensure that a receiver device is available during the test. I would recommend to habe a look to the tests of Presentation API in [2] and the current Test Report in [3].

PS: when you are writing tests please keep in mind that you are testing the API Implementation and NOT the underlying protocols.

Honry commented 7 years ago

@louaybassbouss, really appreciate for your exhaustive sharing, I agree with you that tests should be strictly compliance with API specification without any dependence. I will dig into tests of Presentation API as a reference to write tests for Remote Playback API.

anssiko commented 7 years ago

@Honry, we're likely to discuss this topic at our TPAC F2F on Monday 6 November, so any input, testing-related issues, even if preliminary, prior to that would be appreciated.

Honry commented 7 years ago

Can we talk about the possibility to implement Testing API(quite similar to https://github.com/w3c/presentation-api/issues/440) to break through test automation?

Honry commented 7 years ago

Considering the Remote Playback API is much more simple then Presentation API, that is the amount of manual tests is much less. But implement a Testing API would bring much more efforts.

louaybassbouss commented 7 years ago

@Honry please let me know if can help on this

markafoltz commented 7 years ago

If the test API is strictly about testing the RemotePlayback interface, it could be very simple:

<video>.remote.test.deviceAvailable = true to simulate a remote playback device being available for that element, and false to simulate a device not being available.

If we want to test behavior of the media element itself in remote playback scenarios, that could mean creating a fake remote playback device with specific capabilities.

Honry commented 7 years ago

I think this should be enough. @louaybassbouss, what's your opinion?

louaybassbouss commented 7 years ago

@Honry <video>.remote.test.deviceAvailable = true is enough for Observing remote playback device availability but I am not sure how to test the RemotePlayback Connection State. We may need a 'FakePlaybackScreen' similar to this proposal from @mfoltzgoogle. The 'FakePlaybackScreen' can have some additional attributes like playing, paused, etc. to test the transition between the different remote playback connection states. What do you think?

Honry commented 7 years ago

@louaybassbouss, I was thinking using <video>.remote.test.deviceAvailable = true/false to monitor state change should be enough. But yes, I agree with you that we should also test the video state must be align with the remote playback.

markafoltz commented 7 years ago

We do need to be able to test a failure to connect to a remote playback device (step 4 of "Establishing a Connection.")

As far as testing the behavior of the media element, the mandatory requirements in the spec are:

The user agent MUST send media commands to the remote playback device - but how would this be observable from a test?
The media element MUST reflect changes to the remote playback state, which is similar to testing media playback in general.

Let's discuss this further at the F2F.

mounirlamouri commented 7 years ago

Blink is using a method to force update the device availability. Because we run this only in our tests which do not have a Remote Playback backend, I think the WPT method should be different as there will be an idea of forcing the value on top of the backend. Because of this, I would recommend the method to take true, false and null in order to unset the override. A method might make things clearer to (such as setOverrideDeviceAvailability()). Same could apply for connection state.

mounirlamouri commented 4 years ago

@Honry what's the status of the tests for Remote Playback API? It's my understanding that we are only testing non-user gesture and non-device specific parts of the API which isn't a very large amount. Are there plans to have a WPT API to mock a device?

Honry commented 4 years ago

Oops, I missed this issue. @mounirlamouri, thank you for pinging.

Defining a WebDriver extension API is a more standardized way and which has already been adopted in Generic Sensor, Permissions API, Reporting API, etc.. Moreover, Presentation API also advocated it at https://github.com/w3c/presentation-api/issues/440#issuecomment-342687008.

@anssiko, considering Remote Playback API is CR now, is that possible to roll back it with new WebDriver extension API?

anssiko commented 4 years ago

We can publish a revised CR with WebDriver extension API similarly to Generic Sensor API.

@tidoust for comments.

tidoust commented 4 years ago

@tidoust for comments.

I confirm that the group can publish a revised CR with a WebDriver extension API. In any case, the implementation report needs to cover more than "non-user gesture and non-device specific parts of the API".

Honry commented 4 years ago

Thanks @tidoust, @anssiko for confirming this.

Since I will be OOO next two weeks for Chinese New Year holiday, I will figure out an initial proposal for review after the holiday.

Honry commented 4 years ago

@mounirlamouri, @anssiko, @mfoltzgoogle, @louaybassbouss,

Here is a draft WebDriver based API which I am thinking should be satisfied checkpoints you mentioned before, but I may have missed something, PTAL, all comments are very welcome!

https://github.com/Honry/remote-playback-test-automation/blob/master/webdriver-extension-api.md

One open: how many playback states should be covered?

(Sorry for late due to nCov situation)

anssiko commented 4 years ago

Thanks @Honry! (And no problem for the delay.)

@mfoltzgoogle @mounirlamouri can either of you help review the proposed WebDriver Extension API and give advise on the playback states question?

Honry commented 4 years ago

@mfoltzgoogle, @mounirlamouri, friendly ping. 😃

mounirlamouri commented 4 years ago

Thank you @Honry for the proposal. At a first glance it looks fine but I think we will have to implement and use that in Chrome to give more useful feedback. Do you intend to implement the test api in a browser?

Honry commented 4 years ago

Agree, but I'm not an expert in feature implementation, maybe @anssiko could help looking for someone interested in this. :)

BTW, @mounirlamouri, could you give advise on following open? I am thinking of picking several typical playback states should be sufficient.

One open: how many playback states should be covered?

anssiko commented 4 years ago

I think @foolip knows folks who touch Web Driver bits in Chromium. I don’t think we have someone readily available for this work at the moment. I suggest open a crbug and link it here.

foolip commented 4 years ago

If you'd like to implement https://github.com/Honry/remote-playback-test-automation/blob/master/webdriver-extension-api.md in ChromeDriver, then @johnchen0 should be able to review.

For help testing things in WPT more generally, I would ping @stephenmcgruer @LukeZielinski @Hexcles.

anssiko commented 4 years ago

ACTION: @louaybassbouss to figure out if we want to move forward with test automation or manual testing, to unblock PR publication.

via https://www.w3.org/2020/10/20-webscreens-minutes.html#a01

louaybassbouss commented 3 years ago

Alternative proposal to move forward with Remote Playback Testing using a manual approach. This proposal is not replacing the WebDriver approach discussed in this thread, but just an alternative to Fake Remote Playback device.

We use prepared Mezzanine Content to play on Remote Playback Device. The Mezzanine Content shows QR Code on each video frame which contains meta data about the timing/... . This CTA WAVE project can be used for this purpose.
The Remote Playback Tests are implemented for control device in a way similar to the Presentation API tests. While a Test is running, the screen of the remote playback device will be recorded. This CTA WAVE Project includes tests following this approach.
Test results are generated after the recording is analysed. This CTA WAVE project can be used for this purpose.

Lets discuss this option in our F2F meeting in few minutes.

FritzHeiden commented 2 years ago

The new remote playback tests are now implemented. You can find the pull request here

FritzHeiden commented 1 year ago

The new remote playback tests are now implemented. You can find the pull request here

The pull request is merged and tests are now available in master, however, there is still an open pull request for further style and text changes https://github.com/web-platform-tests/wpt/pull/41852

markafoltz commented 2 months ago

https://github.com/web-platform-tests/wpt/pull/41852 was merged some time ago.

Here is the current WPT dashboard for the automated tests: https://wpt.fyi/results/remote-playback?label=experimental&label=master&aligned

I suppose the next step is to run the manual tests against compatible devices and prepare an implementation report?

anssiko commented 2 months ago

@mfoltzgoogle correct, our publication tracker #130 updated accordingly.

We are now looking for contributors who have compatible devices to run the manual tests:

*-manual.html in https://wpt.live/remote-playback/ (GH repo)

Results can be reported in this issue.

anssiko commented 1 month ago

Updated test results, including for *-manual.html:

https://w3c.github.io/test-results/remote-playback/all.html (diff)

Legend (via https://github.com/w3c/remote-playback/issues/130#issuecomment-2379419389):

ED29: Edge 129 SF17: Safari 17 on macOS SO18: Safari 18 on iOS SP18: Safari 18 on iPad OS

@FritzHeiden thank you for this timely contribution!

w3c / remote-playback

Remote Playback API tests and implementation report #92