Management of conformance files

AOMediaCodec / av1-isobmff

Official specification of the AOM group for the carriage of AV1 in ISOBMFF

https://AOMediaCodec.github.io/av1-isobmff

64 stars 16 forks source link

Management of conformance files #148

Closed cconcolato closed 8 months ago

cconcolato commented 2 years ago

We agree to adopt the following:

We create a conformance folder to store files
Each file is accompanied with a JSON file describing: contributor, license, features exercized, list of Bikeshed ids from asserts in the spec, ...
Use of git LFS to track MP4 files and alike
We automate the generation of a report based on the spec (its assert ids) and on conformance files
- generate errors if an id in the spec does not have matching test file
- generate errors/warnings if a test file does not have a matching id in the spec
- files would be verified against ComplianceWarden.
- file contributors should consider updating their files or the spec if the above generated errors/warnings

cconcolato commented 2 years ago

We could start with the following files: http://download.opencontent.netflix.com.s3.amazonaws.com/index.html?prefix=AV1/

cconcolato commented 2 years ago

We could check with ComplianceWarden if:

they could output a JSON file instead of a text file for further processing
the JSON could include also all assertions that passed (including valid ones), possibly using the Bikeshed-generated ids.

cconcolato commented 1 year ago

@podborski to verify the changes in https://github.com/gpac/ComplianceWarden/issues/35#issuecomment-1356460430

podborski commented 1 year ago

@podborski to verify the changes in gpac/ComplianceWarden#35 (comment)

Assert IDs seem to match the rules from specification text, however they are written only to the error entries and we would need them also for warnings and successful checks. Also some of the rules from cw.exe seem to miss the ID entries.

cconcolato commented 1 year ago

Note that the files marked as Experimental on the Netflix web site should be fine, they are called experimental because they were generated to test the encryption change that Apple requested before the corresponding spec change was agreed, but they should be to-spec now.

podborski commented 1 year ago

@cconcolato Hi Cyril, FYI. the initial version of the script is added to the conformance branch. The idea is to run it like this from the tools directory:

python3 add_files.py -i PathToLocalConformanceFolder -l PathToLicense.txt

It should copy the files into conformance_files dir and add an associated json file to each file.