AOMediaCodec / av1-rtp-spec

Current draft (HTML): https://aomediacodec.github.io/av1-rtp-spec/
Other
63 stars 24 forks source link

Add support for rid simulcast in Dependency Descriptor #200

Closed murillo128 closed 3 years ago

murillo128 commented 3 years ago

Currently the DD is codec agnostic, but does not cover the use case for simulcast when sent over multiple SSRCs.

The main requirement for supporting it would be to be able to have the similar mechanism as the active layers mask but for simulcast streams. This will allow fast switching to a different simulcast stream when the current forwarded one is deactivated.

This is a long lasting request for webrtc developers so I think it would be a great opportunity to cover it and increase the adoption of the DD: https://bugs.chromium.org/p/webrtc/issues/detail?id=5207#c46

If we don't do it, the risk is that an alternative solution is proposed for that use case that potentially lowers the DD usage.

We would need to transmit in each separate simulcast streams the following info:

The change could look like this:


extended_descriptor_fields() {
  template_dependency_structure_present_flag = f(1)
  active_decode_targets_present_flag = f(1)
  simulcast_info_present_flag = f(1)
  custom_dtis_flag = f(1)
  custom_fdiffs_flag = f(1)
  custom_chains_flag = f(1)

  if (template_dependency_structure_present_flag) {
    template_dependency_structure()
    active_decode_targets_bitmask = (1 << DtCnt) - 1
  }

  if (active_decode_targets_present_flag) {
    active_decode_targets_bitmask = f(DtCnt)
  }

  if (simulcast_info_present_flag) {
    simulcast_stream_index = f(5)
    active_active_simulcast_streams_bitmask _present = f(1)
    if (active_active_simulcast_streams_bitmask _present) {
      simulcast_stream_cnt_minus_one = f(5)
      SimulcastSreamCnt = simulcast_stream_cnt_minus_one + 1
      active_active_simulcast_streams_bitmask = (1 << DtCnt) - 1
    }
  }
}
aboba commented 3 years ago

Thanks for raising this issue, Sergio. We had previously examined whether it was possible to provide simulcast information within the bitstream in the multiple SSRC case, and concluded that this did not make sense. However, this information needs to be present somewhere, or else an SFU will find itself wondering whether it is not receiving packets on a stream because of loss, or because the endpoint stopped sending on that stream.

StephenBotzko commented 3 years ago

While this is an important problem to solve, it isn't clear that the best solution is to add it to the DD extension.