search

web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
4.88k stars 3.05k forks source link

/media-source/mediasource-duration.html is disabled in Mozilla & Edge and flaky in Chromium #10920

Open zcorpan opened 6 years ago

zcorpan commented 6 years ago

http://bocoup.github.io/wpt-disabled-tests-report/

Investigate what's up with this test:

Path Products Results Bugs
/media-source/mediasource-duration.html mozilla chromium edge disabled [ Failure Pass ] disabled https://bugzilla.mozilla.org/show_bug.cgi?id=1154016 https://crbug.com/689781 None

(Edit: fix bug links)

zcorpan commented 6 years ago

@acolwell @jyavenard @wolenetz any thoughts on how to fix this test?

jyavenard commented 6 years ago

https://bugzilla.mozilla.org/show_bug.cgi?id=1301409 is for a different test.

The actual explanation on why Mozilla disabled the test was explained there: https://bugzilla.mozilla.org/show_bug.cgi?id=1154016#c138

The test is invalid, due to various problems:

zcorpan commented 6 years ago

Thanks @jyavenard! Do you have suggestions for a fix? Would it help to split up the test to multiple tests? Also I don't know what "pts vs dts Bug" means. :)

jyavenard commented 6 years ago

@zcorpan : pts vs dts is I believe bug https://bugs.chromium.org/p/chromium/issues/detail?id=398130 which is now marked as fixed. One of the assumption in this wpt is that after doing an appendBuffer of the content, the buffered range will contain an interval and that interval will start at 0. However, the mp4 file used for the test doesn't start at 0. For this current test to succeed, it relies on Chromium's pts bug.

As explained in my earlier comment, the test is also highly timing dependent, when I modified the assumptions based buffered range and duration value, it would intermittently fail on slow machines.

zcorpan commented 6 years ago

Would it be an improvement to change the mp4 file to actually start at 0?

jyavenard commented 6 years ago

Well, we also use that file to also ensure in another test that mse implementations correctly calculate buffered range and detect files not starting at 0 (they are very common in the mp4 world) Having a quick look at the modified version, it looks okay to me. Do you have a website hosting this version so I can do further tests?

zcorpan commented 6 years ago

http://w3c-test.org/submissions/10989/media-source/mediasource-duration.html

jyavenard commented 6 years ago

Ok. The test fail on Firefox when checking the various new durations. and it fails for good reasons.

The test assume that when calling sourcebuffer.remove(reducedDuration, fullDuration) then the new duration will now be exactly reduceDuration, or that when playback stops, currentTime will be equal to reducedDuration.

Those are incorrect assumptions. Here reducedDuration is fullDuration / 2.0; unless this time value fall exactly on the end of a frame, the new duration will not be reducedDuration. By spec, remove never removes a partial frame: https://w3c.github.io/media-source/#sourcebuffer-coded-frame-removal Coded Frame Removal Algorithm step 3: "Remove all media data, from this track buffer, that contain starting timestamps greater than or equal to start and less than the remove end timestamp. "

So if the frame of a track buffer that contains reducedDuration timestamp has a start time lesser than the start value, it won't be removed. And the new duration will be the end timestamp of that frame: and that will be greater than reducedDuration.

Even in the case where the pts vs dts bug was present in chrome, the likelihood that truncatedDuration falls exactly at the end of a frame is rather remote, so I can only assume that chrome isn't performing the Coded Frame Removal algorithm as specified. @wolenetz could you confirm?

zcorpan commented 6 years ago

The current version of the test is what Chromium has (outside external/wpt/), and it is believed to be non-flaky. Since my main goal is fixing the flakiness, I would like to merge this and follow up with other fixes separately. I've filed https://github.com/w3c/web-platform-tests/issues/11005

jyavenard commented 6 years ago

The test may not be flaky, but it's wrong to start with.. what is the point of merging it then? Is a test designed to test spec conformance or just the behaviour of a particular engine? If it's to test spec conformance, which part is it testing?

zcorpan commented 6 years ago

It should test spec conformance, but we can get there in two steps instead of one step. :) If you like I can add a warning to the test, linking to #11005. Does that seem OK to you? (Edit: now added.)

The point of this issue is to enable the test for Mozilla and Edge, and mark as non-flaky for Chromium.

zcorpan commented 6 years ago

@wolenetz assigning this to you for now since you're owner of https://bugs.chromium.org/p/chromium/issues/detail?id=689781