validating that test content has correct values/attributes - AVC

[jpiesing]

The DASH validator will check that test content is valid CMAF. It won't check that an individual test stream has the correct values/attributes. We should have a script that uses openly available tools to read the key values from a test stream and then a second script that checks that each stream has the correct values.

See

https://github.com/cta-wave/Test-Content-Generation/issues/50 https://github.com/cta-wave/Test-Content-Generation/issues/49 https://github.com/cta-wave/Test-Content-Generation/issues/47

This issue is specifically for AVC but we should have the same for each codec.

[rbouqueau]

The DASH validator will check that test content is valid CMAF. It won't check that an individual test stream has the correct values/attributes. We should have a script that uses openly available tools to read the key values from a test stream and then a second script that checks that each stream has the correct values.

Thanks for raising it again. We now experience how important it is since the last set of test vectors seems affected by a regression in this!

[nicholas-fr]

An initial script to validate the test content is available here for now: https://github.com/nicholas-fr/test-content-validation/

I tested the 15_30_60 vectors to start with and found issues with the following characteristics in some of the test streams:

• Picture timing SEI presence
• VUI timing information presence
• p-frames only expected
• Sample entry type
• In-band parameter set presence
• CMAF header parameter set presence
• Fragment composition

Some of these may be due to issues with the script of course, due to a spec misunderstanding or a bug, so feedback is welcome.

[rbouqueau]

Thank you. The script was used and proved useful!

[nicholas-fr]

I downloaded all the latest test vectors (dated 2022-09-30) and ran the updated validation script. I can confirm that all files now pass.

## Summary of test results:
#  - Total Pass: 1134
#  - Total Fail: 0
#  - Total Not Tested: 0
#  - Total Not Testable: 81  --> cmaf_initialisation_constraints
#  - Total Not Applicable: 81  --> codec_tier not applicable for AVC

I had a question regarding the test flagged as "not testable" (see issue #43).

[gitwjr]

Oct 17th release passes. Rerun with segment validation with the conformance tool.

[nicholas-fr]

Re-ran the validation script on the latest 2022-10-17 test vectors including DASH-IF conformance checks (CMAF, CTA WAVE, and Segment Validation) and all streams passed:

### SUMMARY OF TEST RESULTS:
#
#  DASH conformance check using https://github.com/Dash-Industry-Forum/DASH-IF-Conformance
#  CLI: php Process_cli.php --cmaf --ctawave --segments <MPD location>
#  - Total Conformance Pass: 81
#  - Total Conformance Fail: 0
#  - Total Conformance Unknown: 0
#
#  WAVE test content definition conformance check:
#  - Total Pass: 1134
#  - Total Fail: 0
#  - Total Not Tested: 0
#  - Total Not Testable: 81
#  - Total Not Applicable: 81

test-content-validation_test-results_2022-11-10.zip

So the only remaining open point is the test flagged as "not testable" (see issue https://github.com/cta-wave/Test-Content/issues/43).

[nicholas-fr]

I believe this issue can now be closed. The remaining open point depends on the creation of new tests and test content and is covered by issue #43.

[gitwjr]

Closed. See above comment.