Recording framerate of video streams in each Representation

[jgkim0518]

Hello. I have a question about recording the framerate in MPD.

I am streaming two videos(UltraHD@50fps and FullHD@25fps) using the HEVC codec through the shaka packager. Its MPD content is as follows:

<AdaptationSet id="0" contentType="video" maxWidth="3840" maxHeight="2160" frameRate="90000/3600" par="16:9">
  <Representation id="0" bandwidth="24188775" codecs="hvc1.1.2.L153" mimeType="video/mp4" sar="1:1" width="3840" height="2160">
    <SegmentTemplate timescale="90000" initialization="video/0/video.mp4" media="video/0/$Time$.m4s" startNumber="112996">
      <SegmentTimeline>
        <S t="50847757200" d="450000" r="8"/>
      </SegmentTimeline>
    </SegmentTemplate>
  </Representation>
  <Representation id="1" bandwidth="8069319" codecs="hvc1.1.2.L120" mimeType="video/mp4" sar="1:1" width="1920" height="1080">
    <SegmentTemplate timescale="90000" initialization="video/1/video.mp4" media="video/1/$Time$.m4s" startNumber="112995">
      <SegmentTimeline>
        <S t="50847350400" d="450000" r="8"/>
      </SegmentTimeline>
    </SegmentTemplate>
  </Representation>
</AdaptationSet>

As you can see, AdaptationSet@framerate is output as 90000/3600. It seems that the framerate of FullHD was recorded. But this is wrong. The framerate of UltraHD is 50fps.

In the DASH-IF-IOP-v4.3 it says: For any Representation within a Video Adaptation Sets as defined in 3.2.13 the following attributes shall be present: @frameRate, if not present in AdaptationSet element

From this, I expect to be able to record the framerate in Representation. How can I record framerate in Representation?

[kqyang]

There may be a problem in detecting the frame rate of the UltraHD stream. What happens if you pass the UltraHD stream only to Shaka Packager?

If the frame rate is still set to 25fps, please record a few minutes of the UltraHD stream and share with us so we can try to re-produce the problem locally. (You can send it to shaka-packager-issues@google.com if it cannot be shared publicly).

[jgkim0518]

Thank you for your answer. Is there any way to record framerate in Representation other than Adaptation Set? The problem isn't that packager can't calculate framerate of UltraHD video stream. The problem is that two video streams(Representations) with the same codec, different resolutions and different framerates are placed under the one Adaptation Set, so that either framerate is not recoded in the MPD. Refer to DASH-IF-IOP-v4.3, I expect to be able to record the framerate in Representation. How can I record framerate in Representation?

Based on the input video stream, maxFrameRate="90000/3600" in Adaptation Set, frameRate="90000/1800" maxPlayoutRate="50" in UltraHD Representation, frameRate="90000/3600" maxPlayoutRate="25" in FullHD Representation should be recorded.

My command is as follows in=udp://xxx.xxx.xxx.xxx:xxx,stream=video,init_segment=package/video/0/video.mp4,segment_template=package/video/0/$Time$.m4s,drm_label=HD in=udp://xxx.xxx.xxx.xxx:xxx,stream=video,init_segment=package/video/1/video.mp4,segment_template=package/video/1/$Time$.m4s,drm_label=HD

[kqyang]

@jgkim0518 Sorry for not explaining our algorithms in the previous comment. Yes, that is supported and it is expected to happen automatically. The current logic is that: if the framerate for all the representations are the same, it is set in the AdaptationSet, otherwise it is set in the individual representations.

So have you confirmed that the frame rate is set correctly if you feed the UltraHD stream? If that is the case, then there is a bug in the logic above; if not, it is a problem in the detection of the frame rate for your UltraHD stream.

[jgkim0518]

@kqyang Please check the e-mail I sent to shaka-packager-issues@google.com. Thanks.

[cosmin]

Here is how frame rates are represented in the manifest if they are different, for example taking a 30fps and 60fps input

<AdaptationSet id="0" contentType="video" width="720" height="1280" maxFrameRate="15360/256" par="9:16">
      <SupplementalProperty schemeIdUri="urn:mpeg:mpegB:cicp:TransferCharacteristics" value="1"/>
      <Representation id="0" bandwidth="1643647" codecs="avc1.64001f" mimeType="video/mp4" sar="1:1" frameRate="15360/512">
        <BaseURL>out_30_dash.mp4</BaseURL>
        <SegmentBase indexRange="894-949" timescale="15360">
          <Initialization range="0-893"/>
        </SegmentBase>
      </Representation>
      <Representation id="1" bandwidth="1661568" codecs="avc1.640020" mimeType="video/mp4" sar="1:1" frameRate="15360/256">
        <BaseURL>out_60_dash.mp4</BaseURL>
        <SegmentBase indexRange="894-961" timescale="15360">
          <Initialization range="0-893"/>
        </SegmentBase>
      </Representation>
    </AdaptationSet>

You can see that frameRate is set at the `Representation` level in this case, and the `maxFrameRate` is set the `AdaptationSet` level. If frame rates were the same for all representations in an adaptation set then `frameRate` would be stored just once at the `AdaptationSet` level.