Review of AV1 RTP Payload Specification: Are any changes needed?

[aboba]

On April 8, 2024 the AV1 CODEC WG met to review potential updates to the AV1 specification. The slides discussed are here: AV1_Layer-specific_OBUs_2024-04-08.pdf

To retain backward compatibility, slide 15 makes the following recommendation:
"New decoders should not enforce the new requirements on obu_extension_flag."

This raised the question: "If the new requirements cannot be enforced, why change the spec at all?"

If the proposed changes are not made, are any changes needed to the AV1 RTP payload specification?

In my first pass review, there didn't seem to be any aspect of the AV1 RTP Payload specification that explicitly depends on the proposed bitstream changes. Rather, there are statements relating to layerIDs within an RTP payload that would apply to any RTP payload specification supporting scalability (e.g. prohibitions against mixing of layers).

[aboba]

@eleft @DanilChapovalov @Philipel-WebRTC PTAL.

[DanilChapovalov]

I've checked the motivation for the change (as described in https://github.com/AOMediaCodec/av1-spec/pull/280) Current AV1 spec wording allows frame with spatial_id=1 to be before frame with spatial_id=0.

In the RTP spec I think we implicitly rely on frames in the same temporal unit to be in spatial id order. So if AV1 spec changes are not approved, we might want to restrict RTP spec to only handle bitstream where frames in same temporal unit are ordered by spatial id. e.g. it might be surprising to set marker bit on last frame with spatial_id=0 when there was also a frame with spatial_id=1.

If " new requirements on obu_extension_flag" are not enforced, then sequence header OBU may have extension, and RTP spec might then need different wording to describe how to handle it, e.g. should specify that OBU extension header should be removed for the SH by the sender and ignored by the receiver.

[mhoro]

Danil, Thank you for reviewing the RTP spec and for reporting your findings.

Everyone, I'd like to float the idea of removing the publication of the AV1 spec update as a dependency on the publication of the RTP for AV1 specification. To that end, I propose that we wait a few weeks for further comments and update the RTP for AV1 text to address the reported issues/comments.

 -michael

On Tue, Apr 9, 2024 at 6:21 AM DanilChapovalov @.***> wrote:

I've checked the motivation for the change (as described in AOMediaCodec/av1-spec#280 https://github.com/AOMediaCodec/av1-spec/pull/280) Current AV1 spec wording allows frame with spatial_id=1 to be before frame with spatial_id=0.

In the RTP spec I think we implicitly rely on frames in the same temporal unit to be in spatial id order. So if AV1 spec changes are not approved, we might want to restrict RTP spec to only handle bitstream where frames in same temporal unit are ordered by spatial id. e.g. it might be surprising to set marker bit on last frame with spatial_id=0 when there was also a frame with spatial_id=1.

If " new requirements on obu_extension_flag" are not enforced, then sequence header OBU may have extension, and RTP spec might then need different wording to describe how to handle it, e.g. should specify that OBU extension header should be removed for the SH by the sender and ignored by the receiver.

— Reply to this email directly, view it on GitHub https://github.com/AOMediaCodec/av1-rtp-spec/issues/238#issuecomment-2044798837, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIUGZQBN7Y7GRZVBNAIQGDDY4PFLVAVCNFSM6AAAAABF5RFPGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBUG44TQOBTG4 . You are receiving this because you were assigned.Message ID: <AOMediaCodec/av1-rtp-spec/issues/238/2044798837 @.*** .com>

[aboba]

FYI, there will be another AV1 update meeting on Monday, April 15. Here are the slides: AV1_Layer-specific_OBUs_2024-04-15.pdf

Here is Adrian's most recent update:

"We have made some progress in studying the pending AV1 specification changes, and have agreed a plan for moving forward.

Review by a couple of people familiar with the AV1 RTP binding specification concluded that, with the addition of a few minor clarifications, the publication dependency of that spec on the pending AV1 spec changes could be overcome, and it can be published independently of any AV1 spec update.

The potential need to publish the pending AV1 spec changes remains, to bring clarification on certain aspects of multilayer coding. But it was concluded that we should wait for discussion of Apple's proposal to add support for multiview coding to AV1 (expected to be discussed in the next 2-3 weeks) before moving forward, since it claims to address some of the outstanding issues.

Other changes that are awaiting publication (CSP_CENTER, cropping rectangle) will be finalized and the expectation is that the AV1 update will be published later this year. If discussion of the new AV1 proposals takes longer than expected, we can always revisit the decision and publish the clarifications separately.

Please let me know if you have any questions, suggestions or concerns.

Best regards, Adrian"

[mhoro]

Update: at the April 15 meeting mentioned in the comment above, attendees reached consensus that, with small modifications, the RTP spec could be published independently of an AV1 spec update. Consequently, it was decided that there was no rush to make any changes to the AV1 spec.

Caveat: I was unable to attend the April 15 meeting and would appreciate confirmation (and correction if necessary) to the summary above from someone who did attend.

[mhoro]

My apologies for the spam. It appears Bernard's previous message captured the update.

In the interest of moving forward with the publication of the AV1 for RTP spec, I propose that interested parties meet to craft a plan to identify and draft text for the required RTP spec modifications. I further propose that we coordinate this effort by some means other than git-hib comments (e.g., email). Who would be interested in participating is this meeting/effort?

[aboba]

I am interested.On Apr 25, 2024, at 09:26, mhoro @.***> wrote:Re: [AOMediaCodec/av1-rtp-spec] Review of AV1 RTP Payload Specification: Are any changes needed? (Issue #238)My apologies for the spam. It appears Bernard's previous message captured the update.In the interest of moving forward with the publication of the AV1 for RTP spec, I propose that interested parties meet to craft a plan to identify and draft text for the required RTP spec modifications. I further propose that we coordinate this effort by some means other than git-hib comments (e.g., email). Who would be interested in participating is this meeting/effort?—Reply to this email directly, view it on GitHub or unsubscribe.You are receiving this email because you authored the thread.Triage notifications on the go with GitHub Mobile for iOS or Android.