search

cta-wave / WMAS

Test suite for the Web Media API Snapshot Specification
Other
18 stars 9 forks source link

Some readyState related tests fail when executed on a device with limited decoding capability #57

Closed jsierant-at-vewd closed 4 years ago

jsierant-at-vewd commented 4 years ago

I've encountered a problem with some tests related to readyState on a device with limited decoding capability:

Each HTML file contains two tests with a corresponding media element set to autoplay. When the file is loaded by a browser the tests are executed in parallel and as a result both media elements try to start playing. On a device with only one decoder there is a race for resources as HTML does not specify which media shall be played first. One of the tests fails because readyState is not set as expected for the corresponding media element. 

The tests pass when run separately (one of tests deleted from HTML file) and on devices without such limitations.

Have you considered such a situation? Can execution be parameterized in a way that tests are run sequentially?

louaybassbouss commented 4 years ago

Thanks @jsierant-at-vewd for reporting this. Each of the test files you mentioned contains 1 video and 1 audio element. This shouldn't be a problem. 2 video elements in playing state may cause issues on platforms that only allow a video element to be in playing state at a time (e.g. HbbTV). I run the 3 test files you mentioned on TV (in HbbTV mode) and all tests pass as you can see in the screenshots below.

I am not sure if the sequential execution of tests in a single file will solve the issue since the playback of both elements starts automatically via the autoplay attribute. PS: The tests and the Testharness JS library are developed within the W3C WPT project.

What do you think?

jsierant-at-vewd commented 4 years ago

I've encounter the problem on a device which can play an one media at a time (in this case it is either an audio or an audio/video media). It's a hardware/decoder limitation.

It is possible that in case of HbbTV you've checked the limitation is only software based as HbbTV specification allow only a single video to be played at a time. So it is a bit different case.

I've mentioned sequential test execution support because with simple resource management (e.g. each new media player takes over the ownership of the decoder causing the previous media to stop/pause) the tests execution could be predictable as there will be no race condition on devices with such a limitation.

Now both tests are triggered at the same time. A source is loaded to each HTML media element and due to the autoplay attribute both attempt to start playing. Removing autoplay would not help as some of the tests require a player to enter playing state (autoplay would have to be replaced by some other similar mechanism).

IMHO the goal of this tests is not to verify if both audio and video media element can enter expected readyState when executed in parallel (which is not a problem when HW allows this) and there are merged into an one HTML file by convenience as there are checking the same thing but for a different HTML media element type.

It seems that a lot of work is required to rewrite all the HTML related files just to separate tests.

Is it possible to add parameterized sequential test execution support?

jpiesing commented 4 years ago

I'd expect an HbbTV implementation to have the same issue - only one media element (either video/audio) with a hardware decoder allocated at one time. Hence in the description given, one media element will get the hardware decode and one won't.

louaybassbouss commented 4 years ago

Thanks @jsierant-at-vewd and @jpiesing for your comments. I think we need a clear statement in the WAVE Specification about using multiple video/audio elements on the same page. The HbbTV Spec is clear about this but the HTML Spec doesn't say anything about this behavior. @johnluther @JohnRiv what is your opinion on this?

@jsierant-at-vewd do you know how many test files use audio and video elements in the same test? If only the 3 test files you mentioned are affected, I think the easiest solution is to split each test into 2 files (1 for video and one for audio) and add the current tests on the blacklist to ignore them while running the tests.

jsierant-at-vewd commented 4 years ago

I'm not sure how many tests are affected. I've encountered problems only with those but more may be affected as this is a race condition kind of problem.

johnluther commented 4 years ago

@louaybassbouss two video elements playing at the same time seems unlikely in our full-screen OTT use case, but I would support adding a note to the spec to caution people, framed in terms of a web app, not a page.

louaybassbouss commented 4 years ago

@johnluther the test cases mentioned above include 1 video and 1 audio element. regarding 2 video elements playing at the same time, I think one relevant use case is picture-in-picture with 1 video playing in fullscreen and 1 video in small size playing on top.

I agree with you to add a note to spec. We may need (maybe in WAVE Test Suite 2019) to identify tests (in WPT) with more than 1 media element and rewrite/split them into files each with only 1 media element. @jsierant-at-vewd this will also solve your issue.

johnluther commented 4 years ago

Thanks for the clarification, @louaybassbouss .

@JohnRiv , what do you think about adding a note?

JohnRiv commented 4 years ago

I agree a note makes sense.

Also FYI in case anyone here wasn't already aware, there is a PIP API in the works in the W3C: https://w3c.github.io/picture-in-picture/

JohnRiv commented 4 years ago

@jsierant-at-vewd we added a PR to the spec (see above). Please let us know if that makes sense to you. We're looking to merge it into the codebase in a week.

jsierant-at-vewd commented 4 years ago

Yes, it summarizes the problem I've encountered.

JohnRiv commented 4 years ago

This issue is closed with the merge of https://github.com/w3c/webmediaapi/pull/241