Exiting because of bad sps

thatdevsherry commented 1 year ago

Hi, I tried out your crate to check if it works with my outdoor camera. It exits out because of bad sps.

Is this a problem with how the camera implements RTSP?

Thank you

Output

I20221015 14:37:53.671 main client::mp4] Using h264 video stream
I20221015 14:37:53.686 main client::mp4] No suitable audio stream found
E20221015 14:37:55.589 main client] Fatal: bad sps

conn: 192.168.10.15:53286(me)->192.168.10.10:554@2022-10-15T14:37:53
stream: TCP, interleaved channel ids 0-1
ssrc: 00000000
seq: 00000068
pkt: 150505@2022-10-15T14:37:54

thatdevsherry commented 1 year ago

Maybe this can provide some context

     Running `target/debug/client info --url 'rtsp://192.168.10.10' --print-streams --print-sdp`
SDP:
v=0
s=streamed by the macro-video rtsp server
t=0 0
a=control:*
a=range:npt=0-
a=x-qt-text-nam:streamed by the macro-video rtsp server
c=IN IP4 0.0.0.0
m=video 0 RTP/AVP 96
b=AS:500
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=TQAf;packetization-mode=1;sprop-parameter-sets=J00AH+dAKALdgKUFBQXwAAADABAAAAMCiwEAAtxoAAIlUX//AoA=,KO48gA==
a=control:track1

stream 0:
Stream {
    media: "video",
    control: Some(
        "rtsp://192.168.10.10/track1",
    ),
    encoding_name: "h264",
    rtp_payload_type: 96,
    clock_rate: 90000,
    channels: None,
    framerate: None,
    depacketizer: Ok(
        Depacketizer(
            H264(
                Depacketizer {
                    input_state: New,
                    pending: None,
                    parameters: Some(
                        InternalParameters {
                            generic_parameters: VideoParameters {
                                rfc6381_codec: "avc1.4D001F",
                                pixel_dimensions: (
                                    1280,
                                    720,
                                ),
                                pixel_aspect_ratio: Some(
                                    (
                                        1,
                                        1,
                                    ),
                                ),
                                frame_rate: Some(
                                    (
                                        2,
                                        40,
                                    ),
                                ),
                                extra_data: Length: 53 (0x35) bytes
                                0000:   01 4d 00 1f  ff e1 00 26  27 4d 00 1f  e7 40 28 02   .M.....&'M...@(.
                                0010:   dd 80 a5 05  05 05 f0 00  00 03 00 10  00 00 03 02   ................
                                0020:   8b 01 00 02  dc 68 00 02  25 51 7f ff  02 80 01 00   .....h..%Q......
                                0030:   04 28 ee 3c  80                                      .(.<.,
                            },
                            sps_nal: b"'M\0\x1f\xe7@(\x02\xdd\x80\xa5\x05\x05\x05\xf0\0\0\x03\0\x10\0\0\x03\x02\x8b\x01\0\x02\xdch\0\x02%Q\x7f\xff\x02\x80",
                            pps_nal: b"(\xee<\x80",
                        },
                    ),
                    pieces: [],
                    nals: [],
                },
            ),
        ),
    ),
    state: Uninit,
}

I20221015 14:49:18.468 main client] Done

scottlamb commented 1 year ago

What make/model/version camera is this?

Is this a problem with how the camera implements RTSP?

Likely yes, but I'm happy to add workarounds as needed.

The SDP you quoted looks fine, as far as I can tell. The SPS there is parseable. And if there were an error in that SDP, Retina should have logged a warning Ignoring bad H.264 format-specific-params: ... and proceeded anyway.

Instead, I think it's struggling with the "in-band parameters" (ones sent in the middle of the RTP data stream, rather than in the SDP). Not sure yet what to do about that. Maybe we could try making it ignore them if it already has valid parameters (as it seems to in this case). We can also improve the logging here to give some more info than bad sps.

It'd help me to see the whole data stream. Would you be able/willing to gather a packet capture with e.g. Wireshark? It could be this Retina run, or maybe better yet some other software talking successfully with this camera (rather than bailing with an error here). The packet capture will have your camera's MAC and such embedded in it, so If you don't want to share it publicly, emailing it to me would be perfectly fine.

thatdevsherry commented 1 year ago

What make/model/version camera is this?

It's a generic V380 camera.

Would you be able/willing to gather a packet capture with e.g. Wireshark?

Sure. Let me check if I can capture some relevant data. I have a raspberry pi setup to stream camera's RTSP using VLC.

thatdevsherry commented 1 year ago

I've mailed the wireshark capture to you. Please let me know if it's what you were looking for.

I did the following:

Switched to v0.4.2, since v0.4.3 exits with bad sps and can never seem to start running
Tried running retina

I noticed this error pop up a couple of times during startup, as well as exiting in middle of receiving packets (v0.4.2 starts up after a couple of tries like old carburetor cars :p)

E20221015 22:44:40.071 main client] Fatal: SPS NAL is 120954 bytes long; must fit in u16

Whenever it does record to out.mp4, VLC doesn't really show any video during playback.

EDIT: I also noticed that the error pops up on v0.4.2 only when I try to stream the HD stream. When selecting to play the lower resolution SD stream, it starts up and doesn't exit. I can't play the recorded out.mp4 on VLC again though.

Also on v0.4.3,bad sps still shows up when trying to play the lower res stream.

thatdevsherry commented 1 year ago

I also noticed when trying to stream HD track, VLC doesn't immediately start showing video. VLC itself starts up but the video comes up a bit after.

I don't have knowledge about how the communication works, but maybe it has to do with VLC waiting to get those "in-band params" and after it gets them, it figures out how to stream video.

scottlamb commented 1 year ago

The packet capture helped. I have in my working copy a program that chews through it, with improved errors. Here's what it says:

bad sps (invalid RBSP byte 0x0 in state TwoZero): Length: 64850 (0xfd52) bytes
0000:   27 4d 00 1f  e7 40 28 02  dd 80 a5 05  05 05 f0 00   'M...@(.........
0010:   00 03 00 10  00 00 03 02  8b 01 00 02  dc 68 00 02   .............h..
0020:   25 51 7f ff  02 80 00 00  00 01 28 ee  3c 80 00 00   %Q........(.<...
0030:   00 01 25 b8  20 00 11 6f  f6 28 f7 5b  50 80 62 e2   ..%. ..o.(.[P.b.
0040:   a0 b9 68 83  02 d1 30 9a  fc 4b 95 1f  84 d3 37 1e   ..h...0..K....7.
0050:   42 44 a7 f8  3f 4b 54 06  21 d3 53 41  55 c2 8d 92   BD..?KT.!.SAU...
0060:   f3 44 dd 5a  ee 24 4d ad  cb 41 31 dc  e4 8e 07 d6   .D.Z.$M..A1.....
0070:   52 fd 05 d5  d4 3d 79 4f  d3 85 36 06  33 57 ec 32   R....=yO..6.3W.2
0080:   fe 5c b8 5c  4d 5f 4f 57  aa 29 dc d7  39 e9 3c 98   .\.\M_OW.)..9.<.
0090:   2c 2d 89 ad  43 98 cf df  6a 85 34 12  9c d5 bc 3f   ,-..C...j.4....?
00a0:   ca 69 39 5b  8a 0a f1 c9  43 ad 70 5a  e3 87 5e aa   .i9[....C.pZ..^.
00b0:   45 fa 7a 95  cb 6c 12 07  c7 9e d6 ae  e1 02 5d e0   E.z..l........].
00c0:   c6 d8 a7 9f  3a eb cb 61  94 b1 7a 00  9c 1b 7d d9   ....:..a..z...}.
00d0:   39 16 04 d3  bf 14 45 54  50 3c 2b d1  21 b0 e0 1b   9.....ETP<+.!...
00e0:   a0 cc 25 30  e4 22 b8 72  27 96 83 8d  b7 1c 89 58   ..%0.".r'......X
00f0:   f4 d8 65 bd  2c d9 8d 93  37 dd 62 3e  47 78 54 f8   ..e.,...7.b>GxT.
...64594 (0xfc52) bytes not shown..

Some background.

H.264 streams are divided into messages called "NAL"s (Network Abstraction Layer, which is a weird name for what it is IMHO). A NAL can be encoded in different ways, including:

As RBSP (raw byte sequence payload).
As an encoded NAL. This has to follow certain rules. It can't have two zero bytes followed by a zero, one, two, or three. That's described a bit here. (You can also download the spec (see here for help finding AV specs), although it's...dense reading.)

The H.264 spec defines an encoding (often called "Annex B format") that uses these sequences of zeros to mark the ends of NALs. This is why they're not allowed to appear inside a single encoded NAL.

RTSP uses another standard called RTP for its data messages. Different audio/video encodings each have their own RFC to describe how RTP transports them. For H.264, it's RFC 6184. It includes:

"Single NAL Unit Packet": one RTP packet = one encoded NAL
FU-A: "Fragmentation Unit A": several RTP packets = one (large) encoded NAL, which is useful when the NAL is very large. RTP packets often are limited to the max size that fits in an unfragmented UDP packet, and they can never be more than 65,536 bytes. "Slice" NALs (ones that have the actual video data) can be much larger than that, so they almost always end up inside FU-As with high-quality video.
"Aggregation Packets": one RTP packet = several NALs, encoded in a special way. Sometimes the SPS and PPS end up together in one aggregation packet. Some cameras use this; others don't.

It looks like your camera is sending FU-As which contain several encoded NALs:

at byte 0, an SPS (sequence parameter set) NAL starts.
at byte 0x26, there's zeros marking a boundary.
at byte 0x2a, a PPS (picture parameter set) starts.
at byte 0x2e, there's another boundary.
at byte 0x32, a slice (actual video data) starts.

This isn't what the standard says to do. Retina's behavior recently changed:

in version v0.4.2,
- when parsing the SPS, it would tolerate extra data at the end of the SPS. It would only complain if the whole FU-A is larger than the parameters are allowed to be inside the AvcDecoderConfigurationRecord. That's the must fit in u16 message you see sometimes, more with the higher-quality stream.
- editing to add: when writing the .mp4, things get weird. I'm not surprised it doesn't play in VLC.
  - it'd write the AvcDecoderConfigurationRecord with what it thinks is just the SPS/PPS but maybe actually has a whole IDR frame inside it. Every time there's what (it thinks) is a new SPS/PPS, it starts a new "sample description" with a new AvcDecoderConfigurationRecord. Depending on how often your camera writes parameters to the video stream and how your mp4 player works, that might produce huge files, weird effects like the first frame reappearing every few seconds, etc.
  - and for video frames, .mp4 is supposed to break apart the NALs by putting in a length prefix. Retina's example would write a .mp4 with a length prefix before what it thinks is a single NAL but is actually several in Annex B format. Maybe some video players convert to Annex B before feeding into a H.264 decoder anyway so things work out. Some probably error out.
in version v0.4.3, I upgraded h264_reader to a version that's stricter. Parsing the SPS complains all the time about the extra data at the end.

To support your camera well, we need to alter Retina's src/codec/h264.rs to everywhere it's currently expecting a single NAL, expect an Annex B stream of multiple NALs, and break them apart. This should allow your camera to work properly: no Retina errors, and the example will write valid .mp4 files. It also should be basically harmless for cameras that follow the standard properly (just a tiny bit of extra CPU).

I could work on this. I also see that you have some experience working with networking in Rust. Would you prefer to try it yourself? I'm happy to walk you through it if so.

thatdevsherry commented 1 year ago

Thanks for the detailed response Scott. I'd love to work on this. I did work with networking in Rust a while back, but I wouldn't consider my rust skills to be that good :p Just started re-learning rust.

I can try to work on this, with a little bit of hand-holding along the way :D

thatdevsherry commented 1 year ago

Update on the issue.

I've managed to throw some code at it and it is now creating an mp4 file that is working!

I'll create a PR soon and we can review if there's anything we need to add :)

scottlamb / retina

Exiting because of bad sps #68

Output