How to decode h264 with aggregation?

[fayfive]

Some h264 stream contain sps+pps+Iframe together in a FU-A, not sps+pps Iframe, sps+pps+Iframe just has one Mark. Here is the first udp package contain sps+pps+Iframe

                         80 60 f1 ae 5a 58   . hrH...(..`..ZX
0030   c4 ae 29 60 8e 9d 7c 87 4d c0 1f 8d 8d 70 78 08   ..)`..|.M....px.
0040   af 70 80 00 00 03 00 80 00 00 1e 47 84 42 37 00   .p.........G.B7.
0050   00 00 01 68 ce 3c 80 00 00 00 01 65 b8 00 01 00   ...h.<.....e....
0060   00 0f c4 8b 8d c3 96 8c 00 d6 d8 28 21 b6 7a e0   ...........(!.z.
0070   06 2c 8d 92 7e 59 e5 65 19 ff fe c0 5e eb dd f9   .,..~Y.e....^...
0080   a6 e8 c0 df f0 de 86 9e b5 1f 1c 1f ff 22 9d 28   .............".(
0090   52 d7 55 a8 05 70 fa 1b 07 44 29 fe 02 e1 f0 e9   R.U..p...D).....
00a0   27 07 3f f0 fb be 75 ad 6b ac 80 32 b1 46 64 20   '.?...u.k..2.Fd 
00b0   d5 c2 18 65 8f f6 ff 87 75 fd ad 6c 60 64 da 61   ...e....u..l`d.a
00c0   14 fb 80 89 eb 00 dc 66 01 f3 38 93 27 59 ef ed   .......f..8.'Y..
00d0   c2 90 8d 40 40 eb d7 ff 5c fc 69 43 83 11 81 ac   ...@@...\.iC....
00e0   1c 94 a8 21 85 81 56 ff ff 88 c7 d5 a4 cd ff 87   ...!..V.........
00f0   f1 c2 83 87 68 18 08 88 ef e6 0a 27 69 f9 15 1b   ....h......'i...
0100   79 91 01 b4 fc ff ff c3 67 48 00 91 49 e3 9c c9   y.......gH..I...
0110   8d d2 25 e5 a1 06 b5 fc 8f 88 89 56 9f 10 3d c9   ..%........V..=.
0120   f1 5d a7 c5 71 7f c3 fe 18 11 01 60 04 97 a7 fb   .]..q......`....
0130   6a f0 03 8f 2e 4c fb d4 80 ac 4c a6 ff a7 d2 94   j....L....L.....
0140   20 ae f0 40 aa e7 80 2c 22 48 91 2c e6 42 e5 32    ..@...,"H.,.B.2
0150   d9 56 11 36 54 c7 b0 34 05 6d c9 1e ce 98 da be   .V.6T..4.m......
0160   46 06 b2 7a 07 85 3c 25 63 a5 e6 75 de a2 99 27   F..z..<%c..u...'
0170   85 f4 7f 4f 8b b3 6e f3 8b 01 a0 2d 39 2c 3c 19   ...O..n....-9,<.
0180   41 20 4f a4 a6 f8 fd 2b ff d3 7b ba 34 46 35 4a   A O....+..{.4F5J
0190   17 7e ef cc f9 3c 16 c4 3c 97 a1 ef e8 ef b0 79   .~...<..<......y
01a0   80 b8 61 62 8c 2f fe 7f 5f bc 77 03 28 2a dc ef   ..ab./.._.w.(*..
01b0   a6 09 e7 92 2d df f4 ff 0d 8a 07 12 5c b0 0d 5a   ....-.......\..Z
01c0   63 a5 4c 2a a3 1c 63 9f bf 3c 22 03 9c 08 07 4c   c.L*..c..<"....L
01d0   f4 f8 0e 70 20 0a 85 e2 22 40 1e a9 88 02 d6 74   ...p ..."@.....t
01e0   66 05 98 3a ce 6b 7f ff c3 fe 30 74 07 02 2a a5   f..:.k....0t..*.
01f0   33 f6 b0 04 d9 cd 68 ea 6d 59 ff a3 33 02 8f 20   3.....h.mY..3.. 
0200   f0 04 42 5c 98 41 10 01 d3 1d 02 2d f0 01 4f a3   ..B\.A.....-..O.
0210   27 c6 27 0a 56 ad 58 65 76 ae fa f9 76 a9 25 e7   '.'.V.Xev...v.%.
0220   a0 99 b9 f9 fe 32 8b 7f ff f6 7d 4b dc 18 13 d1   .....2....}K....
0230   d5 9f c5 c1 23 dd ba 0f 09 75 cd 91 22 70 43 05   ....#....u.."pC.
0240   fa 69 ed f1 9e 0a c4 a0 18 28 80 d0 06 c4 95 c5   .i.......(......
0250   6b 2c 53 1c a9 e3 d3 30 02 4c f4 12 ce 8e 47 f5   k,S....0.L....G.
0260   cd 59 9a 9b bb 70 5f f0 b4 6b 8a a8 65 fb 7f fd   .Y...p_..k..e...
0270   6e 20 38 24 21 5e 01 ae fe 9f 5f a7 7b 19 84 07   n 8$!^...._.{...
0280   c0 fd 5d 30 28 24 79 48 4e 9f b1 c7 25 fd bf 86   ..]0($yHN...%...
0290   8b b8 2d 7d 40 79 20 00 10 05 4c ac 38 e2 52 0e   ..-}@y ...L.8.R.
02a0   52 bc 55 10 c3 96 49 8c b0 ef d6 bc ff a6 6d 93   R.U...I.......m.
02b0   71 3f a5 59 86 67 d4 22 6b 37 bb c7 4d 3f ed 1e   q?.Y.g."k7..M?..
02c0   1f e5 be 00 b0 02 7e c8 5d 13 63 99 f8 d9 9a 64   ......~.].c....d
02d0   9b 4f 92 d6 13 f3 9f 4b fb 1e b7 d9 3a f5 1c 58   .O.....K....:..X
02e0   02 92 a4 35 05 80 3b 69 a6 fc 4b bc cf 48 03 b3   ...5..;i..K..H..
02f0   b1 28 78 86 03 a3 42 38 87 ad 69 fa 9a ab 63 6d   .(x...B8..i...cm
0300   1f f1 00 8e 32 fd 04 50 81 1f d6 2e eb 5c d4 5a   ....2..P.....\.Z
0310   c0 6c d7 a6 3f 7c 41 1a 62 3f ed c3 53 dc 07 01   .l..?|A.b?..S...
0320   88 49 52 07 e4 a9 df be fe 1e a3 b3 a8 26 10 02   .IR..........&..
0330   07 6b 9a 46 9c 15 e2 88 ab a8 2d ee 25 c1 f8 ca   .k.F......-.%...

[aler9]

Hello, the problem is more clear now, you're claiming that in some RTP/H264 streams there are multiple NALUs inside a single RTP Fragmentation Unit (FU-A), and therefore rtph264.Decoder is unable to parse them.

The question is that theoretically a FU-A RTP packet can't hold more than one NALU at a time, therefore, how is it possible that multiple NALUs are packed inside a single FU-A? how are you able to split the SPS, PPS and IDR NALUs from the packet dump you posted?

[aler9]

oh now i see, Annex-B is used to pack SPS, PPS and IDR inside a single FU-A, you can see the [0 0 0 1] delimiter inside your packet:

124 135 77 192 31 141 141 112 120 8 175 112 128 0 0 3 0 128 0 0 30 71 132 66 55
0 0 0 1 104 206 60 128
0 0 0 1 101 184 0 1 0 0 15 196 139 141...

[aler9]

I've investigated, and the decoder already supports decoding Annex-B inside FU-A packets. This is a working example that uses the packet dump you provided:

https://go.dev/play/p/dGqbXP59Z4k

Why are you having problem decoding such streams?

[fayfive]

I found that if I push the first frame that is Pframe and the next frame is sps+pps+Iframe to the decoder, and the sps+pps+Iframe can not decode. If the first frame is sps+pps+Iframe, it works well.

[aler9]

I found that if I push the first frame that is Pframe and the next frame is sps+pps+Iframe to the decoder, and the sps+pps+Iframe can not decode. If the first frame is sps+pps+Iframe, it works well.

Everything is clear now, the reason is that rtph264.Decoder decodes Annex-B if and only if the first received RTP packet contains an Annex-B delimiter:

https://github.com/aler9/gortsplib/blob/4b7fed505b9a60c9f6a99a6cb99f5c87f1b9cbc3/pkg/formatdecenc/rtph264/decoder.go#L194-L227

The decoder will be improved in order to parse Annex-B even if the first received RTP packet doesn't contain an Annex-B delimiter.

[fayfive]

That means the first frame must be Iframe to the decoder? I modify the gortsplib/pkg/formatdecenc/rtph264/decoder.go

func (d *Decoder) removeAnnexB(nalus [][]byte) ([][]byte, error) { 
    if !d.firstNALUParsed { 
        d.firstNALUParsed = true 

        if len(nalus) == 1 { 
            nalu := nalus[0] 

            i := bytes.Index(nalu, []byte{0x00, 0x00, 0x00, 0x01}) 
            if i >= 0 { 
                d.annexBMode = true 

                if !bytes.HasPrefix(nalu, []byte{0x00, 0x00, 0x00, 0x01}) { 
                    nalu = append([]byte{0x00, 0x00, 0x00, 0x01}, nalu...) 
                } 

                return h264.AnnexBUnmarshal(nalu) 
            } else {
                d.firstNALUParsed = false // I add here to avoid the first frame is Pframe or else
            }
        } 
    } else if d.annexBMode { 
        if len(nalus) != 1 { 
            return nil, fmt.Errorf("multiple NALUs in Annex-B mode are not supported") 
        } 

        nalu := nalus[0] 

        if !bytes.HasPrefix(nalu, []byte{0x00, 0x00, 0x00, 0x01}) { 
            nalu = append([]byte{0x00, 0x00, 0x00, 0x01}, nalu...) 
        } 

        return h264.AnnexBUnmarshal(nalu) 
    } 

    return nalus, nil 
 } 

I have created a PR to you

[github-actions[bot]]

This issue is being locked automatically because it has been closed for more than 6 months. Please open a new issue in case you encounter a similar problem.