AV1Payloader does not work for WebRTC

aleksitto-gh commented 2 months ago

Your environment.

Version: v1.8.6
Browser: Google Chrome, Chromium, Arc

What did you do?

I'm trying to use AV1 codec for WebRTC using the libaom lib. The encoding works fine but probably AV1Payloader is not ready for that yet.

As a result, there is no video in my browser.

I found the browser expects to see obu size info but it is not sent.

[56649:64259:0703/185801.600480:WARNING:video_rtp_depacketizer_av1.cc(313)] Mismatch in obu_size. signaled: 0, actual: 1442
[56649:64259:0703/185801.637214:WARNING:video_rtp_depacketizer_av1.cc(313)] Mismatch in obu_size. signaled: 0, actual: 394
[56649:64259:0703/185801.664048:WARNING:video_rtp_depacketizer_av1.cc(313)] Mismatch in obu_size. signaled: 0, actual: 686
[56649:64259:0703/185801.701495:WARNING:video_rtp_depacketizer_av1.cc(313)] Mismatch in obu_size. signaled: 0, actual: 656
[56649:64259:0703/185801.732393:WARNING:video_rtp_depacketizer_av1.cc(313)] Mismatch in obu_size. signaled: 0, actual: 2044
[56649:64259:0703/185801.769189:WARNING:video_rtp_depacketizer_av1.cc(313)] Mismatch in obu_size. signaled: 0, actual: 276

The obu size flag is set as required via libaom in the obu header, I guess it is a default behavior.

I tried to manually change it as not required for the first encoded byte:

sMask := byte(0b11111101)
encoded[0] &= sMask

Packet buffer logic with the manually changed configuration:

pkt := C.aom_codec_get_cx_data(e.codec, &iter)
if pkt == nil {
break
}

if pkt.kind == C.AOM_CODEC_CX_FRAME_PKT {
...
pktBuf := C.aom_pktBuf(pkt)
pktSize := C.aom_pktSz(pkt)

encoded := C.GoBytes(unsafe.Pointer(pktBuf), pktSize)

sMask := byte(0b11111101)
encoded[0] &= sMask
}

c++ additional functions:

aom_codec_frame_flags_t aom_pktFrameFlags(aom_codec_cx_pkt_t *pkt) {
  return pkt->data.frame.flags;
}
void *aom_pktBuf(aom_codec_cx_pkt_t *pkt) {
  return pkt->data.frame.buf;
}
int aom_pktSz(aom_codec_cx_pkt_t *pkt) {
  return pkt->data.frame.sz;
}

it helped a little bit, the error with obu size mismatch has gone, but I found another errors on the browser side:

[56649:64259:0703/185901.432737:WARNING:video_receive_stream2.cc(831)] No decodable frame in 209975 us requesting keyframe. Last RTP timestamp 473597759.
[56649:64259:0703/185901.653905:WARNING:video_receive_stream2.cc(831)] No decodable frame in 219832 us requesting keyframe. Last RTP timestamp 473615438.
[56649:64259:0703/185901.876031:WARNING:video_receive_stream2.cc(831)] No decodable frame in 220216 us requesting keyframe. Last RTP timestamp 473635687.
[56649:64259:0703/185902.107335:WARNING:video_receive_stream2.cc(831)] No decodable frame in 220999 us requesting keyframe. Last RTP timestamp 473657668.
[56649:64259:0703/185902.322845:WARNING:video_receive_stream2.cc(831)] No decodable frame in 214457 us requesting keyframe. Last RTP timestamp 473675091.
[56649:64259:0703/185902.540952:WARNING:video_receive_stream2.cc(831)] No decodable frame in 216235 us requesting keyframe. Last RTP timestamp 473696303.
[56649:64259:0703/185902.767741:WARNING:video_receive_stream2.cc(831)] No decodable frame in 222425 us requesting keyframe. Last RTP timestamp 473718583.
[56649:64259:0703/185902.979717:WARNING:video_receive_stream2.cc(831)] No decodable frame in 210875 us requesting keyframe. Last RTP timestamp 473735219.

And still, no video is shown.

What did you expect?

AV1 Payload supports WebRTC. Video stream is playing correctly.

What happened?

AV1Payloader does not support obu size info configs for OBU payloads
Not sure exactly but maybe when obu size info is missing the browser can not understand when the keyframe is triggered because this info is not reflected in the AV1Payloader logic.

lebedyncrs commented 5 days ago

@aleksitto-gh tried this one ?

aleksitto-gh commented 5 days ago

@lebedyncrs yes, but it did not work for WebRTC

pion / rtp