search

aboba / hevc-webrtc

Other
5 stars 2 forks source link

Clarify on the VPS/SPS/PPS requirement before key-frame #20

Closed taste1981 closed 2 months ago

taste1981 commented 6 months ago

keyframe(i-frame) in HEVC may be one of the IRAP picture types: IDR, CRA or BLA.

While spec mentions parameter sets must be before key-frames, is this for IDR_N_LP/IDR_W_RADL, or it is actually requiring VPS/SPS/PPS to be also before, for example, CRA?

aboba commented 6 months ago

Since CRA or BLA doesn't completely refresh decoder state (e.g. can reference previous frames), it's not clear to me that prior VPS/SPS/PPS is always required. But do we understand all the situations where the absence of VPS/SPS/PPS would be a problem (e.g. resolution changes)?

taste1981 commented 6 months ago

VPS/SPS/PPS is not necessary for CRA when there's no resolution change. So the asks is to clarify in Spec, that only IDR needs to include sequence parameter set, not all IRAP pictures should include them.

Philipel-WebRTC commented 6 months ago

I think we should update the spec to require SPS/PPS/VPS before any "keyframe" like NALU.

An IDR/CRA/BLA sent MUST always be preceded by the
relevant parameter sets sent in a packet (not necessarily a separate
packet) with the same RTP timestamp as the IDR/CRA/BLA.
aboba commented 6 months ago

This seems like the cleanest approach. If we don't require SPS/PPS/VPS with the same timestamp as a CRA/BLA, then we'd need to analyze when not doing so would cause problems. The following situations seem like they could be problematic:

  1. If the fps in the CRA/BLA changes (FPS = sps:vui:time_scale / sps:vui:num_units_in_tick = vps:time_scale / vps:num_units_in_tick).
  2. If the size of the picture changes (sps:pic_width_in_luma_samples and sps:pic_height_in_luma_samples).
  3. If the aspect ratio changes (sps::vui::aspect_ratio_idc).

But is this a complete list? What if the number of temporal layers changes?

aboba commented 2 months ago

Fixed via PR #21