search

cta-wave / dpctf-tests

Repo for DPCTF Tests. We prefer to keep the Tests separated from Test Runner
Other
1 stars 5 forks source link

switching set test when one track is a higher profile #108

Closed jpiesing closed 1 year ago

jpiesing commented 1 year ago

For AVC, the switching set includes one track at level 4.2 for 1080p50, all the others are level 4.0. 1080p50 is very widely supported but formally not required and only very recently added to the WAVE content spec.

Please find a way to make the switching set test adapt to implementations not supporting AVC level 4.2.

I did think about checking with isTypeSupported if level 4.2 is supported and discarding that CMAF track if isTypeSupported returns false but quite what would be done when playout[] references that track wasn't clear to me. I guess there could be two different playout[], one not including level 4.2 and one including it.

More generic versions exist where the check could be based on how the track is identified in the MPD.

Alternatively we can duplicate the test with a different playout[] but I'd prefer an adaptive test if it's not too ugly or too hacky.

jpiesing commented 1 year ago

From the May 9th meeting. @louaybassbouss noted that cloning the test with the 2 instances having different playout configurations was preferred as it was least work and lowest risk. An adaptive test was possible but required changes to code that is currently shared between tests as well as an additional communication with the OIF. @rcottingham Resillion cannot comment definitively but share @louaybassbouss 's views.

FritzHeiden commented 1 year ago

We decided to go for the duplicate test solution. Duplicate tests are now supported, however, I still need to configure the tests. Can you specify which tests exactly are affected by this? Additionally, I would need to know what representations to use for each of the tests, that is, which representations are level 4.2 and which are 4.0

jpiesing commented 1 year ago

The tests that are impacted are the switching set tests. The only representation that is 4.2 is 1080p50/60. Hence one version of the test would have a playout array including 1080p50/60 and the other version would not.

FritzHeiden commented 1 year ago

This is now implemented. Will be merged with other current changes regarding the new content

FritzHeiden commented 1 year ago

This feature is now merged.

gitwjr commented 1 year ago

@yanj-github to verify. @FritzHeiden to provide the recordings.

FritzHeiden commented 1 year ago

recordings can be found here: https://drive.google.com/file/d/1WDdRKVcvc-SRxHPxRbrLJbKlKjYUBZas/view?usp=sharing

DannyKokkinos commented 1 year ago

Hey @FritzHeiden could you please approve my request to access the recordings from the google drive.

FritzHeiden commented 1 year ago

@DannyKokkinos I updated access permissions for the link. Please try again

DannyKokkinos commented 1 year ago

Hey @FritzHeiden! Thank you for updating the permissions. Could you please tell me which branch of dpctf-tests was used to record the switching_set_tests? The reason I ask is because I cannot find the test ID in the tests.json for any of the active branches.

rcottingham commented 1 year ago

We have tested switching set with our own recording and Observation Framework reports results correctly.

FritzHeiden commented 12 months ago

Hey @FritzHeiden! Thank you for updating the permissions. Could you please tell me which branch of dpctf-tests was used to record the switching_set_tests? The reason I ask is because I cannot find the test ID in the tests.json for any of the active branches.

This was a bug I just pushed a fix for. The test id should now be contained in the tests.json