regular-playback-of-chunked-content__chunked fails currentTime test

cta-wave / dpctf-tests

Repo for DPCTF Tests. We prefer to keep the Tests separated from Test Runner

Other

2 stars 5 forks source link

regular-playback-of-chunked-content__chunked fails currentTime test #158

Closed jpiesing closed 6 months ago

jpiesing commented 8 months ago

One TV fails the cfhd_12.5_25_50-local/regular-playback-of-chunked-content__chunked.html test partly because the the detected value of currentTime remains at zero until frame 7 (inclusive) it shows "ct: 0" until the end of frame 7. Only at frame 8 does the value of ct change - "ct: 0.173".

This is the transition from frame 7 to frame 8.

This could be a bug in many different places;

In the TV software (e.g. not incrementing the JavaScript currentTime property)
In the test case (e.g. missing some updates to the app-generated QR code)
In the OF (if updating the app-generated QR code is somehow impossible)

FritzHeiden commented 7 months ago

We will investigate if this is a bug in the test implementation

FritzHeiden commented 7 months ago

How the overlay works is that its update function is called on different events. For the most recent current_time value, there is a listener for the timeupdate event (docs), so the first time ct is updated in the QR-code depends on the first event from the API.

jpiesing commented 7 months ago

@yanj-github When does the OF expect to detect the QR code from the test? What does not generating this QR code until frame 8 mean for the OR?

yanj-github commented 7 months ago

@yanj-github When does the OF expect to detect the QR code from the test? What does not generating this QR code until frame 8 mean for the OR?

@jpiesing I can look myself but if we see the QR code (not string display) there is additional value "d: x?" this is delay in generating QR code from TR. So OF use this delay to calculate the correct time of status QR code. If you can see "d" with a delay than it should be correct and correctly handled by OF.

yanj-github commented 7 months ago

@jpiesing Do you want me to still investigating? The ct diff on "ct: 0.173" time_diff=107 is longer than others but it PASS test when the tolerance to 1frame+150ms.

jpiesing commented 7 months ago

@jpiesing Do you want me to still investigating? The ct diff on "ct: 0.173" time_diff=107 is longer than others but it PASS test when the tolerance to 1frame+150ms.

I'll re-run the OF with the adjusted tolerances.

yanj-github commented 7 months ago

@jpiesing I have already ran them. You can try on your end if you wish to. The regular-playback-of-chunked-content__chunked.html test on three best devices passed current time check with Allowed tolerance is 1 frames, 150ms. Total failure count is 0. This will be addressed by https://github.com/cta-wave/device-playback-task-force/issues/123.

jpiesing commented 7 months ago

@jpiesing I have already ran them. You can try on your end if you wish to. The regular-playback-of-chunked-content__chunked.html test on three best devices passed current time check with Allowed tolerance is 1 frames, 150ms. Total failure count is 0. This will be addressed by cta-wave/device-playback-task-force#123.

Then this can be closed when #123 is signed off by DPCTF.

jpiesing commented 6 months ago

Closing as https://github.com/cta-wave/device-playback-task-force/issues/123 has been reviewed and agreed in the main DPCTF meeting with all stakeholders present.