low-latency-short-buffer-playback__t2.html failed on 7 of 8 TV sets in London plugfest

jpiesing commented 6 months ago

low-latency-short-buffer-playback__t2.html failed the "[OF] Every video frame S[k,s] shall be rendered and the video frames shall be rendered in increasing presentation time order." observation on 7 of 8 TV sets in the London plugfest.

jpiesing commented 5 months ago

https://github.com/cta-wave/device-playback-task-force/issues/126 applies here.

yanj-github commented 4 months ago

Re; https://github.com/cta-wave/device-playback-task-force/issues/128 and updated spec r2.05:

from https://github.com/cta-wave/dpctf-tests/blob/master/test-config.json "waiting_timeout" is used. If spec changes to use waiting_duration I think the configuration file should changes to match it.

"low-latency-short-buffer-playback.html": { "maximum_forward_buffer": 500, "waiting_timeout": 500 } @FritzHeiden is it easy to update the configuration to match the spec change?

Observation: The video element transitions into playing state before the waiting_duration, timeout elapses. Test runner terminates test when this observation is not met, is it possible to be observed by test runner, does test runner already reporting error for this observation please? If not, I am not sure how OF can make this observation from test runner status QR code? Maybe test runner can report error in status QR code?

FritzHeiden commented 4 months ago

@FritzHeiden is it easy to update the configuration to match the spec change?

Yes it is. Should I go ahead and push this to master?

Observation: The video element transitions into playing state before the waiting_duration, timeout elapses. Test runner terminates test when this observation is not met, is it possible to be observed by test runner, does test runner already reporting error for this observation please? If not, I am not sure how OF can make this observation from test runner status QR code? Maybe test runner can report error in status QR code?

I can add an error event for this

yanj-github commented 4 months ago

@FritzHeiden is it easy to update the configuration to match the spec change?

Yes it is. Should I go ahead and push this to master?

It is up to you. If no regression concerns (it might be used HTML+JS of dpctf-tests) and easy to change please go ahead. If the change has a rick of breaking anything please raise it https://github.com/cta-wave/device-playback-task-force/issues/128 it might be simpler to change the spec now. OF does not use this parameter so we are fine on OF end.

Observation: The video element transitions into playing state before the waiting_duration, timeout elapses. Test runner terminates test when this observation is not met, is it possible to be observed by test runner, does test runner already reporting error for this observation please? If not, I am not sure how OF can make this observation from test runner status QR code? Maybe test runner can report error in status QR code?

I can add an error event for this

Yes please if this can be done. So when OF detects the specific error event reports failure other wise the test will pass.

jpiesing commented 4 months ago

2024-05-14: We need to decide if this can be closed now and, if not, what needs to be done before it can be closed.

jpiesing commented 4 months ago

2024-05-14: Fraunhofer and Resillion will coordinate.

yanj-github commented 4 months ago

@FritzHeiden @louaybassbouss and @jpiesing re; https://github.com/cta-wave/dpctf-tests/issues/153#issuecomment-2100478574:

Can we update "waiting_timeout" to "waiting_duration" in test-config.json please?
or update spec to replace "waiting_duration" with "waiting_timeout"?

Whichever is easier at this stage. All other tests uses "waiting_duration" so option 1 will make the parameter more consistent with others. OF does not read this parameter so no impact on OF end.

yanj-github commented 4 months ago

Also https://github.com/cta-wave/device-observation-framework/issues/82 I have added issue on OF. We are waiting for error event to be added.

FritzHeiden commented 4 months ago

1. Can we update "waiting_timeout" to "waiting_duration" in test-config.json please?

Done with https://github.com/cta-wave/dpctf-tests/pull/180

jpiesing commented 4 months ago

What does this mean for running the OF on recordings from the London plugfest? Will the release version of the OF stop supporting those recordings?

FritzHeiden commented 4 months ago

What does this mean for running the OF on recordings from the London plugfest? Will the release version of the OF stop supporting those recordings?

This parameter is only relevant for executing the test. As Yan already mentioned, the OF ignores this parameter, so this should have no impact on analyzing older recordings.

yanj-github commented 4 months ago

Thanks @FritzHeiden OF does not use this parameter, no OF changes required. and so no impact.

jpiesing commented 4 months ago

Observation: The video element transitions into playing state before the waiting_duration, timeout elapses. Test runner terminates test when this observation is not met, is it possible to be observed by test runner, does test runner already reporting error for this observation please? If not, I am not sure how OF can make this observation from test runner status QR code? Maybe test runner can report error in status QR code?

I can add an error event for this

@FritzHeiden Was the error event added?

FritzHeiden commented 4 months ago

@FritzHeiden Was the error event added?

This was already implemented as an error event, however, it did not help to understand why the test failed. So I removed it as an error and added a dedicated field in the report with https://github.com/cta-wave/dpctf-tests/pull/182

(last entry):

cta-wave / dpctf-tests

low-latency-short-buffer-playback__t2.html failed on 7 of 8 TV sets in London plugfest #153