search

shaka-project / shaka-packager

A media packaging and development framework for VOD and Live DASH and HLS applications, supporting Common Encryption for Widevine and other DRM Systems.
https://shaka-project.github.io/shaka-packager/
Other
2k stars 510 forks source link

feat: Generate the entire AV1 codec string when the colr atom is present #1205

Closed CaitlinOCallaghan closed 1 year ago

CaitlinOCallaghan commented 1 year ago

As per the AV1 spec, the codec string may contain optional color values. These color values are critical for detecting HDR video streams - see Issue 1007 and Issue 1202. This PR extracts the missing color information from the mp4 colr atom, if present, and generates the full AV1 codec string. Writing the colr information to the muxed mp4 will come in a later PR.

References:

Testing

Manual: AV1 video WITH colr atom

Note: Presence of the colr atom in hdr_av1_test.mp4 was verified with mp4dump

  1. Generate Packager MPD with: ./out/Release/packager in=~/Downloads/bbbhdr_av1.mp4,stream=video,output=av1_with_colr.mp4 --mpd_output av1_with_colr.mpd
  2. Check manifest generated by Shaka Packager for Representation codecs string: 
    <?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/shaka-project/shaka-packager version 31129ee-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" minBufferTime="PT2S" type="static" mediaPresentationDuration="PT1.100000023841858S">
<Period id="0">
<AdaptationSet id="0" contentType="video" width="640" height="360" frameRate="15360/512" subsegmentAlignment="true" par="16:9">
  <Representation id="0" bandwidth="778146" codecs="av01.1.01M.10.0.000.09.16.09.0" mimeType="video/mp4" sar="1:1">
    <BaseURL>av1_with_colr.mp4</BaseURL>
    <SegmentBase indexRange="808-851" timescale="15360">
      <Initialization range="0-807"/>
    </SegmentBase>
  </Representation>
</AdaptationSet>
</Period>
</MPD>
  3. Generate FFMPEG MPD with: ffmpeg -i ~/Downloads/bbbhdr_av1.mp4 -c:v copy -an -seg_duration 2 -streaming 1 -utc_timing_url "https://time.akamai.com/?iso" -media_seg_name 'num_$Number%05d$.m4s' -init_seg_name 'num_init.m4s' -use_timeline 0 -f dash av1_with_colr.mpd
  4. Check manifest generated by FFMPEG for Representation codecs string: 
    <?xml version="1.0" encoding="utf-8"?>
<MPD xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:mpeg:dash:schema:mpd:2011"
xmlns:xlink="http://www.w3.org/1999/xlink"
xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 http://standards.iso.org/ittf/PubliclyAvailableStandards/MPEG-DASH_schema_files/DASH-MPD.xsd"
profiles="urn:mpeg:dash:profile:isoff-live:2011"
type="static"
mediaPresentationDuration="PT1.1S"
minBufferTime="PT2.2S">
<ProgramInformation>
    <Title>Big Buck Bunny, Sunflower version</Title>
</ProgramInformation>
<Period id="0" start="PT0.0S">
    <AdaptationSet id="0" contentType="video" segmentAlignment="true" bitstreamSwitching="true" lang="eng">
        <Representation id="0" mimeType="video/mp4" codecs="av01.1.01M.10.0.000.09.16.09.0" bandwidth="775498" width="640" height="360" frameRate="30/1">
            <SegmentTemplate timescale="1000000" duration="2000000" availabilityTimeOffset="1.967" initialization="num_init.m4s" media="num_$Number%05d$.m4s" startNumber="1">
            </SegmentTemplate>
        </Representation>
    </AdaptationSet>
</Period>
<UTCTiming schemeIdUri="urn:mpeg:dash:utc:http-xsdate:2014" value="https://time.akamai.com/?iso"/>
</MPD>
  5. Compare - both manifests have codecs="av01.1.01M.10.0.000.09.16.09.0" :)

Manual: AV1 video WITHOUT colr atom

Note: The lack of the colr atom in av1-BBB.mp4 was verified with mp4dump

  1. Generate Packager MPD with: ./out/Release/packager in=~/Downloads/av1-BBB.mp4,stream=video,output=av1_NO_colr.mp4 --mpd_output av1_NO_colr.mpd
  2. Check manifest generated by Shaka Packager for Representation codecs string: 
    <?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/shaka-project/shaka-packager version 31129ee-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" minBufferTime="PT2S" type="static" mediaPresentationDuration="PT10S">
<Period id="0">
<AdaptationSet id="0" contentType="video" width="1920" height="1080" frameRate="15360/512" subsegmentAlignment="true" par="16:9">
  <Representation id="0" bandwidth="847200" codecs="av01.0.08M.08" mimeType="video/mp4" sar="1:1">
    <BaseURL>av1_NO_colr.mp4</BaseURL>
    <SegmentBase indexRange="805-848" timescale="15360">
      <Initialization range="0-804"/>
    </SegmentBase>
  </Representation>
</AdaptationSet>
</Period>
</MPD>
  3. Generate FFMPEG MPD with: ffmpeg -i ~/Downloads/av1-BBB.mp4 -c:v copy -an -seg_duration 2 -streaming 1 -utc_timing_url "https://time.akamai.com/?iso" -media_seg_name 'num_$Number%05d$.m4s' -init_seg_name 'num_init.m4s' -f dash av1_NO_colr.mp4
  4. Check manifest generated by FFMPEG for Representation codecs string: 
    <?xml version="1.0" encoding="utf-8"?>
<MPD xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:mpeg:dash:schema:mpd:2011"
xmlns:xlink="http://www.w3.org/1999/xlink"
xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 http://standards.iso.org/ittf/PubliclyAvailableStandards/MPEG-DASH_schema_files/DASH-MPD.xsd"
profiles="urn:mpeg:dash:profile:isoff-live:2011"
type="static"
mediaPresentationDuration="PT10.0S"
minBufferTime="PT20.0S">
<ProgramInformation>
    <Title>Big Buck Bunny, Sunflower version</Title>
</ProgramInformation>
<Period id="0" start="PT0.0S">
    <AdaptationSet id="0" contentType="video" segmentAlignment="true" bitstreamSwitching="true" lang="und">
        <Representation id="0" mimeType="video/mp4" codecs="av01.0.08M.08" bandwidth="845199" width="1920" height="1080" frameRate="30/1">
            <SegmentTemplate timescale="15360" initialization="num_init.m4s" media="num_$Number%05d$.m4s" startNumber="1">
                <SegmentTimeline>
                    <S t="0" d="153600" />
                </SegmentTimeline>
            </SegmentTemplate>
        </Representation>
    </AdaptationSet>
</Period>
<UTCTiming schemeIdUri="urn:mpeg:dash:utc:http-xsdate:2014" value="https://time.akamai.com/?iso"/>
</MPD>
  5. Compare - both manifests have codecs="av01.0.08M.08" :)

Unittest

./out/Release/codecs_unittest ./out/Release/mp4_unittest

Format

git clang-format --style Chromium --binary /usr/bin/clang-format 80e024013df87a4bfeb265c8ea83cfa2a0c5db0f

CaitlinOCallaghan commented 1 year ago

Rebased after strange issue with unrelated commits being pulled in.