search

cta-wave / device-observation-framework

This repository contains the WAVE Device Observation Framework. The DPCTF Device Observation Framework determines pass or fail results, based on observations taken of tests which are run on a device by the DPCTF Test Runner. DPCTF is the Device Playback Compatibility Task Force of the CTA WAVE Project (https://CTA.tech/WAVE and standards@CTA.tech for more info).
Other
1 stars 3 forks source link

failing video start-up delay observation should not result in failing other sub-tests #50

Closed jpiesing closed 1 year ago

jpiesing commented 1 year ago

The test specification defines that the observation for video start-up delay with a "should" unlike other observations that use a "shall".

Currently TSMax is configured to 120ms but with a note that "This constraint is defined as a first approach but may be refined after running some initial tests.".

Any value we pick that is not absurd (e.g. 2.0s) is likely to have some implementations somewhere that fail it.

In these circumstances, I believe that failing the video start-up delay observation should not result in failing other observations.

yanj-github commented 1 year ago

Hi @jpiesing each observation are observed separately and failing one observation should not directly cause other observations to fail. "ts_max": 120 is defined in test-config.json file. Choose different value should not cause other observations to fail. I can confirm that "Failing video start-up delay observation should not result in failing other sub-tests". If this is not the case, I would like to have more details about the issue please?

jpiesing commented 1 year ago

Closing after further analysis of the failing OF tests.