low-latency-playback-over-gaps__t2.html fails in all 8 TVs in London plugfest

jpiesing commented 3 months ago

In the London plugfest, low-latency-playback-over-gaps__t2.html failed the following 2 observation in 5 or 6 of 8 TV sets and failed to execute to completion in two more.

OF] Every video frame S[k,s] shall be rendered and the video frames shall be rendered in increasing presentation time order.
[OF] Video: The presented sample shall match the one reported by the currentTime value within the tolerance of +/-(2/framerate + 20ms)

jpiesing commented 3 months ago

Not resuming playback after the gap should be reported as failure rather than TIMEOUT.

FritzHeiden commented 2 months ago

Not resuming playback after the gap should be reported as failure rather than TIMEOUT.

How should we detect this error? As this is unwanted behavior rather than an actual error that is thrown by the application, we would have to manually throw the error so it appears correctly in the results. Here is what comes to my mind: When play position is close enough to the beginning of the gap and the playback remains in waiting state for a certain time, throw an error.

What do you think?

jpiesing commented 2 months ago

The failure of currentTime is fixed by https://github.com/cta-wave/device-playback-task-force/issues/123.

The failure of 'Every video frame' has the following error on TV 2.

FAIL: First frame found is 7, expected to start from 1. First frame number tolerance is 0. Mid frame number tolerance is 10. Frames out of order 368, 251. Last frame detected before gap 120 is within the tolerance of 'stall_tolerance_margin'=7.5 frames of expected frame 125. Total of missing frames is 6.

@yanj-github What is "Frames out of order"? How should that be interpreted?

yanj-github commented 2 months ago

@jpiesing It means frame 251 is detected after frame 368, I would go back to recording and navigate to frame 368 and goes frame by frame. If you have debug logs or the recording that can share with me I can have a look.

jpiesing commented 2 months ago

@louaybassbouss @FritzHeiden Please see my email about sharing recordings from London with Resillion. Both TVs 2 and 4 have this issue.

louaybassbouss commented 2 months ago

@louaybassbouss @FritzHeiden Please see my email about sharing recordings from London with Resillion. Both TVs 2 and 4 have this issue.

Upload in progress. Will send details to @yanj-github.

jpiesing commented 2 months ago

I've labelled this as "Blocker" on the basis that we want to include this test in the release. If this turns out to be more complex than expected then we can drop the test from the release and remove "Blocker".

yanj-github commented 2 months ago

https://github.com/cta-wave/device-playback-task-force/issues/125 I need help on this please.

jpiesing commented 2 months ago

https://github.com/cta-wave/device-playback-task-force/issues/126 also applies here

yanj-github commented 2 months ago

https://github.com/cta-wave/device-playback-task-force/issues/125

The failure of 'Every video frame' has the following error on TV 2.

FAIL: First frame found is 7, expected to start from 1. First frame number tolerance is 0. Mid frame number tolerance is 10. Frames out of order 368, 251. Last frame detected before gap 120 is within the tolerance of 'stall_tolerance_margin'=7.5 frames of expected frame 125. Total of missing frames is 6.

@yanj-github What is "Frames out of order"? How should that be interpreted?

Frame out of order caused by duplication: What I observed from recording is that the playback started from beginning and it stopped at frame 120, then it jumped to frame 251 and played till 368 then jumped back again to frame 251 then played till the end. The duration from frame 251 till 368 is repeated.

Will this be test runner or device issue? The test contains duplicated duration. Frame 251 and frame 368, from the example.

FritzHeiden commented 1 month ago

Frame out of order caused by duplication: What I observed from recording is that the playback started from beginning and it stopped at frame 120, then it jumped to frame 251 and played till 368 then jumped back again to frame 251 then played till the end. The duration from frame 251 till 368 is repeated.

Will this be test runner or device issue? The test contains duplicated duration. Frame 251 and frame 368, from the example.

This is also what I noticed when looking at the recordings. As this test runs properly in the browser and on other TVs, I don't believe its a test runner issue. Here is my guess what is happening: When reaching the beginning of the gap, the test should wait for a specific time until skipping to the end of the gap. Some devices skip over this gap automatically and continue playing the video. As soon as the waiting time is over the test skips to the end of the gap, which in this case results in a seek backwards.

jpiesing commented 1 month ago

Frame out of order caused by duplication: What I observed from recording is that the playback started from beginning and it stopped at frame 120, then it jumped to frame 251 and played till 368 then jumped back again to frame 251 then played till the end. The duration from frame 251 till 368 is repeated. Will this be test runner or device issue? The test contains duplicated duration. Frame 251 and frame 368, from the example.

This is also what I noticed when looking at the recordings. As this test runs properly in the browser and on other TVs, I don't believe its a test runner issue. Here is my guess what is happening: When reaching the beginning of the gap, the test should wait for a specific time until skipping to the end of the gap. Some devices skip over this gap automatically and continue playing the video. As soon as the waiting time is over the test skips to the end of the gap, which in this case results in a seek backwards.

Is it reasonable to make the test JS code detect devices that skip over the gap automatically and report a FAIL at that point?

FritzHeiden commented 1 month ago

Is it reasonable to make the test JS code detect devices that skip over the gap automatically and report a FAIL at that point?

Implemented with https://github.com/cta-wave/dpctf-tests/pull/176

jpiesing commented 1 month ago

Both TVs 2 and 4 have the 'frames out of order' issue which we believe is an incorrect implementation. TV 7 had a TIMEOUT. How do we move this test forwards?

yanj-github commented 1 month ago

@jpiesing if fixes done on test runner I think we need a new recording and run with OF. Maybe we cannot release this due to unable to re-take recording?

FritzHeiden commented 1 month ago

We may run this test on our selection of TVs in the lab and see if the frames out of order error still occurs, or at least is replaced by the new error from the test runner

jpiesing commented 1 month ago

We may run this test on our selection of TVs in the lab and see if the frames out of order error still occurs, or at least is replaced by the new error from the test runner

If the frame out of order does not occur then we can consider this resolved.

jpiesing commented 1 month ago

2024-05-14: Wait to see the results from the Fraunhofer zoo check and close as discussed in the two previous comments.

louaybassbouss commented 1 month ago

This are the results on 3 different TV sets after adding a new assertion in the test to make sure that the video remains in the waiting state. @yanj-github I will share the 3 recordings with you.

yanj-github commented 1 month ago

@louaybassbouss thanks. I will check the recording once I have got them. From the result, it seems to me that all the test ended at frame 251 and the playback only last around for 10 minutes. I don't this this is what we expected. I'd expecting the playback starts from 1 to around 125 and then 251 till 750.

louaybassbouss commented 1 month ago

@yanj-github I just sent the recordings

yanj-github commented 1 month ago

@louaybassbouss from looking at the recording the issue is caused by early reporting status=finished. OF uses status=finished to determine the end of test and make observation. I have noticed the status changed to finished at frame 251. Can test runner stay playing until it is actually finishes the test please?

louaybassbouss commented 1 month ago

Thanks @yanj-github we will check. @FritzHeiden Please check the status reporting of finished state

yanj-github commented 1 month ago

@jpiesing and @louaybassbouss The spec stating two different modes live and VOD. I can see that we only have live mode testing but the VOD mode test is missing.

jpiesing commented 1 month ago

I think there are two cases which are not properly reflected in the DPCTF spec.

Simulated VOD where the app carries on appending data after the gap without waiting, i.e. when the player gets to the gap, there is already data after the gap and
Simulated live where the app delays appending data after the gap until the duration of the gap is over, i.e. when the player gets to the gap, there is no data after the gap

I'm not sure if what we currently have is one of these or something in-between.

FritzHeiden commented 1 month ago

@louaybassbouss from looking at the recording the issue is caused by early reporting status=finished. OF uses status=finished to determine the end of test and make observation. I have noticed the status changed to finished at frame 251. Can test runner stay playing until it is actually finishes the test please?

This is fixed with https://github.com/cta-wave/dpctf-tests/pull/178

jpiesing commented 1 month ago

2028-05-28: This remains open as the DPCTF spec needs updating.

cta-wave / dpctf-tests

low-latency-playback-over-gaps__t2.html fails in all 8 TVs in London plugfest #152