search

scottlamb / retina

High-level RTSP multimedia streaming library, in Rust
https://crates.io/crates/retina
Apache License 2.0
244 stars 48 forks source link

support live555 servers older than 2017.06.04 (eg some Reolink models), which have buggy RTP/TCP #17

Closed scottlamb closed 2 years ago

scottlamb commented 3 years ago

Latest understanding: these cameras are using LIVE555 Streaming Media v2013.04.08 (according to the session-level tool attribute in their DESCRIBE response) which has at least two major bugs described in live555's changelog:

2015.05.03:
- Updated the "RTSPServer" implementation to fix a bug in RTP/RTCP-over-TCP streaming.
  Before, if the "RTSPClientConnection" object closed before the "RTSPClientSession" object,
  and the TCP connection was also being used for RTP/RTCP-over-TCP streaming, then the
  streaming state (in the "RTSPClientSession") would stay alive, even though the TCP socket
  had closed (and the socket number possibly reused for a subsequent connection).
  This could cause a problem when the "RTSPClientSession" was later reclaimed (due to inactivity).
  Now, whenever a "RTSPClientConnection" object is closed (due to the RTSP TCP connection closing),
  we make sure that we also close any stream that had been using the same TCP connection
  for RTP/RTCP-over-TCP streaming.  (Thanks to Kirill Zhegulev for noting this issue.)
2013.12.04:
- Updated the "sendDataOverTCP()" function (in "RTPInterface.cpp") to allow for the possibility of
  one of the "send()" calls partially succeeding - i.e., writing some, but not all, of its data.

also, there was a follow-up to the 2015.05.03 entry two years later:

2017.06.04:
- Fixed a bug in "RTPInterface::removeStreamSocket()" that could cause not all 'TCP stream'
  records for a given socket number to be removed if a TCP socket I/O error occurred
  (during RTP/RTCP-over-TCP streaming).  (Thanks to Gerald Hansink et al for reporting this.)

Currently your best bet to successfully talk with these cameras is to ask the manufacturer for a firmware upgrade (see note below for the Reolink 420-5MP / hardware version IPC_51516M5M) or connect with UDP via retina::client::Transport::Udp (#30). We have some other ideas below.

Original comment:

As mentioned in this Apr 29th comment on a moonfire-nvr issue, Reolink cameras have an odd behavior. They sometimes set up a stream with one ssrc, eg:

RTP-Info: url=trackID=1;seq=40477;rtptime=2077624148;ssrc=c517011f,url=trackID=2;seq=0;rtptime=0;ssrc=00000000

then send RTP packets for both this one and another ssrc. I think the latter is for an earlier failed session or something. Sometimes they'll even send the RTP packets for the other ssrc before the connection is even authenticated, which seems like a security hole (if you expose the cameras to an untrusted network, which is probably a bad idea anyway).

Currently when this happens, Retina will fail with an error such as the following (from one of @jlpoolen 's logs today in this moonfire-nvr issue):

Expected ssrc=Some(21cd90d7) seq=Some(8f4a) got ssrc=6d87df1a seq=88b9 ts=1866533263 (mod-2^32: 1866533263), npt 0.000 at [192.168.1.88:33514(me)->192.168.1.48:554@2021-08-14T11:00:57 pos=1510@2021-08-14T11:00:57]

I think we should add an option to so that when we know what ssrc to expect, we just (log and) ignore packets with any other ssrc. This will likely make these cameras behave acceptably, especially if (as I suspect) the other ssrc goes away after a minute or so when the stale session times out.

jlpoolen commented 3 years ago

A possible tool: a program that uses the Rust rtsp client that simply opens a connection and tallies the packets received and logs anomalies. This, along with other items you may have learned about with other cameras, could give a quality report on the camera's ability to keep a stream going (and the libraries ability to handle error/resets). It could be run outside of moonfire-nvr and left running for a period of time, say up to 24-72 hours. Almost like a nurse taking vitals before a doctor sees a patient: getting a profile of the camera's ability to send packets conforming to the specifications.

jlpoolen commented 3 years ago

I was starting to do something like what you looking into in this issue with ffmpeg and the real server when I got interrupted or distracted. (Having these hardware issues really throws me off track.) Generating logs has taken some time for me to perfect and I'll go the extra distance (i.e. handling & preserving ANSI codes) where a lot of prospective users may not. If there were a simple program that you feed the IP, logon credentials to that would then generate a log on the camera's ability to manage an rtsp feed and its resilience (and that of the network) could help crystallize issues that need consideration or handling and provide a stream-lined path of zeroing in on issues.

scottlamb commented 3 years ago

This specific issue is in theory fixed in Retina. I'm going to fix another Reolink problem I found on my own test camera then prepare a Moonfire NVR change to pick up both.

Re: tool: did you see this comment? It's a little bit like what you're talking about. But I'm not super motivated right now to polish it into something that gets detailed spec compliance info automatically. I have more improvements I want to make than time already, so I'm not going spend a lot of time trying to to get detailed bug reports from people who don't care enough to gather them. I'm prioritizing getting Retina and Moonfire NVR working really well on cameras I have access to (including polished on-NVR and on-camera analytics, live view, and scrub bar UI) over getting it working on every camera out there.

scottlamb commented 3 years ago

I found a live555 changelog entry about this:

2015.05.03:
- Updated the "RTSPServer" implementation to fix a bug in RTP/RTCP-over-TCP streaming.
  Before, if the "RTSPClientConnection" object closed before the "RTSPClientSession" object,
  and the TCP connection was also being used for RTP/RTCP-over-TCP streaming, then the
  streaming state (in the "RTSPClientSession") would stay alive, even though the TCP socket
  had closed (and the socket number possibly reused for a subsequent connection).
  This could cause a problem when the "RTSPClientSession" was later reclaimed (due to inactivity).
  Now, whenever a "RTSPClientConnection" object is closed (due to the RTSP TCP connection closing),
  we make sure that we also close any stream that had been using the same TCP connection
  for RTP/RTCP-over-TCP streaming.  (Thanks to Kirill Zhegulev for noting this issue.)

The problematic Reolink cameras say a=tool:LIVE555 Streaming Media v2013.04.08 so no surprise that they're affected by this bug.

My workaround isn't working well: at the timeout of the original session, the camera seems to be sending bytes which aren't not properly framed as RTSP interleaved data, we're dropping the connection, and the cycles repeats.

Another option is to just ensure we send a TEARDOWN anywhere possible, and then wait out the session timeout any time it's not possible.

jlpoolen commented 3 years ago

I'm updating the ip 48 GarageWest camera which is:

 model: RLC-420-5MP
 Hw No.: IPC_51516M5M

and if successfully accomplished, I will test with the Retina client example.

jlpoolen commented 3 years ago

You have to be careful in selecting which upgrade package. Firmware number is not sufficient, you also must align to the model number of your camera. I mistakenly downloaded the first firmware match only read the PDF within the zip archive that the model that upgrade applied to did not match my model. I then returned to Reolink's site and found several matches to my firmware, and thus discerned which one applied to my camera model. I applied the patch and rebooted. Will test this evening with Retina.

scottlamb commented 3 years ago

Here's the cause of the framing error in the old live555 code:

Boolean RTPInterface::sendDataOverTCP(int socketNum, u_int8_t const* data, unsigned dataSize, Boolean forceSendToSucceed) {
  if (send(socketNum, (char const*)data, dataSize, 0/*flags*/) != (int)dataSize) {
    // The TCP send() failed.

    if (forceSendToSucceed && envir().getErrno() == EAGAIN) {
      // The OS's TCP send buffer has filled up (because the stream's bitrate has exceeded the capacity of the TCP connection!).
      // Force this data write to succeed, by blocking if necessary until it does:
#ifdef DEBUG_SEND
      fprintf(stderr, "sendDataOverTCP: resending %d-byte send (blocking)\n", dataSize); fflush(stderr);
#endif
      makeSocketBlocking(socketNum);
      Boolean sendSuccess = send(socketNum, (char const*)data, dataSize, 0/*flags*/) == (int)dataSize;
      makeSocketNonBlocking(socketNum);
      return sendSuccess;
    }
    return False;
  }

  return True;
}

send can partially send the data, which happens when the TCP window fills. This code sends the whole thing from the beginning when that happens, which is wrong. This bug has also since been fixed:

2013.12.04:
- Updated the "sendDataOverTCP()" function (in "RTPInterface.cpp") to allow for the possibility of
  one of the "send()" calls partially succeeding - i.e., writing some, but not all, of its data.

(btw, live555's latest code is still bad. If one client's TCP window fills, the entire server will stall waiting for it to come back.)

scottlamb commented 3 years ago

I flashed a RLC-420-5MP, hardware version IPC_51516M5M with the 12/11/2020 firmware, aka v3.0.0.136_20121101. It still says a=tool:LIVE555 Streaming Media v2013.04.08. So this was ancient history for live555 but is a current problem for Reolink cameras.

Here are some options I see:

  1. complain to Reolink support that the latest available firmware for these models has badly flawed support for RTSP/TCP and ask them to update the live555 version. (Unless they have negotiated a special license with live555, they are legally required to comply in countries that care about licenses. But Reolink is Chinese, and I don't think China is such a country...) [edit: Reolink came through and prepared new firmware for the RLC-420-5MP. See below.]
  2. switch to RTP/UDP. Normally I'd expect RTP/UDP to be flakier and fussier than RTP/TCP but with these bugs it's likely the reverse. Retina doesn't support RTP/UDP today but it's something I'd like to add eventually. [edit: done! #30]
  3. don't use Reolink's RTSP support at all. Instead use their proprietary protocol (see Neolink) or RTMP. Via an RTSP bridge, or in Moonfire's case I could eventually add direct support for these. [edit: Neolink says "Neolink does not support other cameras such as the RLC-420, since they already provide native RTSP." But maybe they'd be open to adding this support if we point out that Reolink's native RTSP really sucks.]
  4. avoid the additional ssrc via teardown (when possible) or waiting 65 seconds (when not). Hope that without the redundant data, the framing error doesn't come up enough to be a big problem. (Another idea I tried, discarding the extra interleaved data, didn't work out well. With Reolink cameras the stale session apparently never times out if it's sending data on a still-active session. So it really needs teardown or a cooling-off period.) [edit: Retina 0.3.1 implements this idea. It still has some rough edges; see #34.)
  5. do 4 and also make an ugly, old-live555-specific hack to avoid the framing error also: before interpreting an interleaved data packet, wait for the following message. If the second message doesn't parse, figure out how many redundant bytes were added mid-packet by the sendDataOverTCP() bug and strip them out before parsing the first message.
scottlamb commented 3 years ago

I just wrote to Reolink.

jlpoolen commented 3 years ago

or

  1. Declare Reolink cameras unsupported at this time.

I'd be interested to learn what other manufacturers use as servers and their conformance to standards. It seems having Scott spend time on one brand which has been documented to be using a buggy server has diminishing returns.

I've been using Reolink because they're cheap and if they get stolen or vandalized, it's not going to be expensive to replace. If other brands provide a lot less problems conforming to standards, then it would be appropriate to weigh their cost. If Reolinks cost $40, and Brand X costs $70 and works well within the RTSP protocol, then it may make sense to switch brands. Now, if the next least expensive brand that plays nicely costs $150, then it might merit further consideration.

I think further time Scott spends on Reolink's issues is time away from other enhancements that affect all users of moonfire-nvr.

scottlamb commented 3 years ago

I'm currently not sure what manufacturers to recommend, though. :-(

jlpoolen commented 3 years ago

I flashed my older one as described above with the correct firmware version for the model I had. I, too, have the session (using the LIVE555 openRTSPclient) showing:

v=0
o=- 1629795645777963 1 IN IP4 192.168.1.48
s=Session streamed by "preview"
i=h264Preview_01_main
t=0 0
a=tool:LIVE555 Streaming Media v2013.04.08
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session streamed by "preview"
a=x-qt-text-inf:h264Preview_01_main
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:500
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=640033;sprop-parameter-sets=Z2QAM6zoAoALWQ==,aO48MA==
a=control:trackID=1
m=audio 0 RTP/AVP 97
c=IN IP4 0.0.0.0
b=AS:256
a=rtpmap:97 MPEG4-GENERIC/16000
a=fmtp:97 streamtype=5;profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1408; profile=1;
a=control:trackID=2

Opened URL "rtsp://192.168.1.48:554/h264Preview_01_main", returning a SDP description:
v=0
o=- 1629795645777963 1 IN IP4 192.168.1.48
s=Session streamed by "preview"
i=h264Preview_01_main
t=0 0
a=tool:LIVE555 Streaming Media v2013.04.08
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session streamed by "preview"
a=x-qt-text-inf:h264Preview_01_main
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:500
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=640033;sprop-parameter-sets=Z2QAM6zoAoALWQ==,aO48MA==
a=control:trackID=1
m=audio 0 RTP/AVP 97
c=IN IP4 0.0.0.0
b=AS:256
a=rtpmap:97 MPEG4-GENERIC/16000
a=fmtp:97 streamtype=5;profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1408; profile=1;
a=control:trackID=2

I tried several sessions using the Retina client and continued to have problems. The best session would be about 16 seconds and then I'd get an error message. Examples:

282699224 (mod-2^32: 282699224), npt 11.968: 249-byte audio frame
652994919 (mod-2^32: 652994919), npt 11.900: 29782-byte video frame
282700248 (mod-2^32: 282700248), npt 12.032: 249-byte audio frame
652997889 (mod-2^32: 652997889), npt 11.933: 30061-byte video frame
653000859 (mod-2^32: 653000859), npt 11.966: 30276-byte video frame
E20210824 17:30:42.477 main client] Fatal: [192.168.1.28:48462(me)->192.168.1.48:554@2021-08-24T17:30:29, 12688689@2021-08-24T17:30:42] RTSP framing error: Invalid RTSP message; next 128 of 790 buffered bytes are:
0000:   0b 32 1a 41  68 15 23 ea  79 50 6c e3  d1 f7 6f a2   .2.Ah.#.yPl...o.
0010:   6e 61 82 e0  2a db df 8c  d4 7a 3d 11  fa 38 3e c5   na..*....z=..8>.
0020:   9d 02 84 cb  49 5e c4 d4  bb b0 25 16  57 50 24 b6   ....I^....%.WP$.
0030:   76 98 91 26  60 90 ff a2  d5 01 da c2  ce d2 56 5f   v..&`.........V_
0040:   79 39 a5 02  b5 ae 9c 92  44 3e 42 2d  72 28 6d 84   y9......D>B-r(m.
0050:   2b a1 8e 5d  cc 8d 89 23  29 05 f7 7a  4e d8 48 5b   +..]...#)..zN.H[
0060:   ed d3 07 43  a1 7f 25 9b  b4 6f e1 90  4d 96 37 61   ...C..%..o..M.7a
0070:   45 f7 1d 6a  ad d4 4c cc  3b c2 45 35  21 c8 27 f3   E..j..L.;.E5!.'.
I20210824 17:30:42.478 main client] Done
Tue Aug 24 05:30:42 PM PDT 2021
jlpoole@taurus /usr/local/src/retina $ 

4240106907 (mod-2^32: 4240106907), npt 15.932: 32993-byte video frame
4240109877 (mod-2^32: 4240109877), npt 15.965: 33618-byte video frame
4061414557 (mod-2^32: 4061414557), npt 16.000: 249-byte audio frame
E20210824 17:46:18.304 main client] Fatal: [192.168.1.28:48544(me)->192.168.1.48:554@2021-08-24T17:46:02, 16728945@2021-08-24T17:46:18] RTerror: Invalid RTSP message; next 128 of 1447 buffered bytes are:
0000:   60 50 14 fc  bb 08 c3 9e  00 10 96 5c  81 9a 24 23   `P.........\..$#
0010:   bf 15 b9 c5  d4 34 85 69  ca 20 c9 ff  14 4b b9 d9   .....4.i. ...K..
0020:   fd 7d 87 af  c8 26 67 63  71 c4 88 01  09 27 09 7d   .}...&gcq....'.}
0030:   37 67 9c 86  1c c3 94 a2  1b ac 34 dd  69 4f 70 d5   7g........4.iOp.
0040:   ac d6 e3 2a  33 f2 2a 25  0f 03 54 d0  e5 9c 90 10   ...*3.*%..T.....
0050:   cc 3b 98 c1  99 01 7a ed  ee d9 fb ca  93 f9 ad 40   .;....z........@
0060:   2e 3c 5f 29  46 bf ed 78  eb ad b4 a5  b2 27 ee cf   .<_)F..x.....'..
0070:   58 82 7a 24  1b 35 c0 b7  1f 94 de 74  dc 85 2f 08   X.z$.5.....t../.
I20210824 17:46:18.304 main client] Done
Tue Aug 24 05:46:18 PM PDT 2021
jlpoole@taurus /usr/local/src/retina $ date; cargo run --example client mp4 --url rtsp://192.168.1.48:554/h264Preview_01_main --username moon --password fire123 /tmp/retina_out.mp4;date;

I had to wait several minutes or I would get receiving data packets type errors which is line with Scott's analysis.

Now, I did successfully save a 10 minute video (with audio) using the Live555 openRTSP client. There may be blips, but it continued on. Here's what was printed to my console during the 10 session:

Created new TCP socket 3 for connection
Connecting to 192.168.1.48, port 554 on socket 3...
...remote connection opened
Sending request: OPTIONS rtsp://192.168.1.48:554/h264Preview_01_main RTSP/1.0
CSeq: 2
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)

Received 152 new bytes of response data.
Received a complete OPTIONS response:
RTSP/1.0 200 OK
CSeq: 2
Date: Wed, Aug 25 2021 02:21:10 GMT
Public: OPTIONS, DESCRIBE, SETUP, TEARDOWN, PLAY, PAUSE, GET_PARAMETER, SET_PARAMETER

Sending request: DESCRIBE rtsp://192.168.1.48:554/h264Preview_01_main RTSP/1.0
CSeq: 3
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)
Accept: application/sdp

Received 175 new bytes of response data.
Received a complete DESCRIBE response:
RTSP/1.0 401 Unauthorized
CSeq: 3
Date: Wed, Aug 25 2021 02:21:10 GMT
WWW-Authenticate: Digest realm="LIVE555 Streaming Media", nonce="435a79a4baaee7fe09baf6278cfe05ce"

Resending...
Sending request: DESCRIBE rtsp://192.168.1.48:554/h264Preview_01_main RTSP/1.0
CSeq: 4
Authorization: Digest username="moon", realm="LIVE555 Streaming Media", nonce="435a79a4baaee7fe09baf6278cfe05ce", uri="rtsp://192.168.1.48:554/h264Preview_01_main", response="892c0c789b008ce2d51853d8c85a81df"
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)
Accept: application/sdp

Received 890 new bytes of response data.
Received a complete DESCRIBE response:
RTSP/1.0 200 OK
CSeq: 4
Date: Wed, Aug 25 2021 02:21:10 GMT
Content-Base: rtsp://192.168.1.48/h264Preview_01_main/
Content-Type: application/sdp
Content-Length: 717

v=0
o=- 1629795645777963 1 IN IP4 192.168.1.48
s=Session streamed by "preview"
i=h264Preview_01_main
t=0 0
a=tool:LIVE555 Streaming Media v2013.04.08
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session streamed by "preview"
a=x-qt-text-inf:h264Preview_01_main
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:500
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=640033;sprop-parameter-sets=Z2QAM6zoAoALWQ==,aO48MA==
a=control:trackID=1
m=audio 0 RTP/AVP 97
c=IN IP4 0.0.0.0
b=AS:256
a=rtpmap:97 MPEG4-GENERIC/16000
a=fmtp:97 streamtype=5;profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1408; profile=1;
a=control:trackID=2

Opened URL "rtsp://192.168.1.48:554/h264Preview_01_main", returning a SDP description:
v=0
o=- 1629795645777963 1 IN IP4 192.168.1.48
s=Session streamed by "preview"
i=h264Preview_01_main
t=0 0
a=tool:LIVE555 Streaming Media v2013.04.08
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session streamed by "preview"
a=x-qt-text-inf:h264Preview_01_main
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:500
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=640033;sprop-parameter-sets=Z2QAM6zoAoALWQ==,aO48MA==
a=control:trackID=1
m=audio 0 RTP/AVP 97
c=IN IP4 0.0.0.0
b=AS:256
a=rtpmap:97 MPEG4-GENERIC/16000
a=fmtp:97 streamtype=5;profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1408; profile=1;
a=control:trackID=2

Created receiver for "video/H264" subsession (client ports 35696-35697)
Created receiver for "audio/MPEG4-GENERIC" subsession (client ports 56590-56591)
Sending request: SETUP rtsp://192.168.1.48/h264Preview_01_main/trackID=1 RTSP/1.0
CSeq: 5
Authorization: Digest username="moon", realm="LIVE555 Streaming Media", nonce="435a79a4baaee7fe09baf6278cfe05ce", uri="rtsp://192.168.1.48/h264Preview_01_main/", response="d463e3d308152052aa01ed9305469a97"
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)
Transport: RTP/AVP;unicast;client_port=35696-35697

Received 203 new bytes of response data.
Received a complete SETUP response:
RTSP/1.0 200 OK
CSeq: 5
Date: Wed, Aug 25 2021 02:21:10 GMT
Transport: RTP/AVP;unicast;destination=192.168.1.28;source=192.168.1.48;client_port=35696-35697;server_port=6970-6971
Session: 54C4E3AC

Setup "video/H264" subsession (client ports 35696-35697)
Sending request: SETUP rtsp://192.168.1.48/h264Preview_01_main/trackID=2 RTSP/1.0
CSeq: 6
Authorization: Digest username="moon", realm="LIVE555 Streaming Media", nonce="435a79a4baaee7fe09baf6278cfe05ce", uri="rtsp://192.168.1.48/h264Preview_01_main/", response="d463e3d308152052aa01ed9305469a97"
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)
Transport: RTP/AVP;unicast;client_port=56590-56591
Session: 54C4E3AC

Received 203 new bytes of response data.
Received a complete SETUP response:
RTSP/1.0 200 OK
CSeq: 6
Date: Wed, Aug 25 2021 02:21:10 GMT
Transport: RTP/AVP;unicast;destination=192.168.1.28;source=192.168.1.48;client_port=56590-56591;server_port=6972-6973
Session: 54C4E3AC

Setup "audio/MPEG4-GENERIC" subsession (client ports 56590-56591)
Outputting to the file: "stdout"
Sending request: PLAY rtsp://192.168.1.48/h264Preview_01_main/ RTSP/1.0
CSeq: 7
Authorization: Digest username="moon", realm="LIVE555 Streaming Media", nonce="435a79a4baaee7fe09baf6278cfe05ce", uri="rtsp://192.168.1.48/h264Preview_01_main/", response="7dce274e8dc69e172c45fc9125d88472"
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)
Session: 54C4E3AC
Range: npt=0.000-600.000

Received 251 new bytes of response data.
Received a complete PLAY response:
RTSP/1.0 200 OK
Server: Rtsp Server/2.0
CSeq: 7
Date: Wed, Aug 25 2021 02:21:10 GMT
Range: npt=0.000-
Session: 54C4E3AC
RTP-Info: url=trackID=1;seq=8202;rtptime=718423480;ssrc=d89a16ec,url=trackID=2;seq=51872;rtptime=3942782428;ssrc=6aca52b4

Started playing session
Receiving streamed data (for up to 600.000000 seconds)...
Sending request: TEARDOWN rtsp://192.168.1.48/h264Preview_01_main/ RTSP/1.0
CSeq: 8
Authorization: Digest username="moon", realm="LIVE555 Streaming Media", nonce="435a79a4baaee7fe09baf6278cfe05ce", uri="rtsp://192.168.1.48/h264Preview_01_main/", response="00429a724edcc215d201530ed4392596"
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)
Session: 54C4E3AC

Received 65 new bytes of response data.
Received a complete TEARDOWN response:
RTSP/1.0 200 OK
CSeq: 8
Date: Wed, Aug 25 2021 02:31:10 GMT

jlpoole@taurus /usr/local/src/Live/live/testProgs $
scottlamb commented 3 years ago

It looks like live555/openRTSP is just skipping all the bytes it doesn't understand. The frame before will be corrupt (because it has some duplicated data in place of some fresh data), and it might do weird things as it interprets video/audio data as headers. I'd prefer to avoid that if possible. Not only does it feel gross, but it could mask other problems (including bugs in Retina itself), and it'd be a security problem in some contexts (the equivalent of an HTTP desync attack). I think if we add a workaround for the bad framing, I'd prefer to do one more narrowly focused on this bug. Or just avoid the issue by adding UDP support and recommending UDP with these cameras.

jlpoolen commented 3 years ago

what about having three modes in Retina? 1) as-is - croaks on certain anomolies as it does now, 2) tries to continue, but logs problem for retrospective analysis, 3) dangerous mode --- imitates ffmpeg & live and does what it can to continue saving MP4 files without error reporting? I like the 2nd mode if it could have a log file that might be reviewed every 24-48 hours that could be processed with a grep command to give statistics such as 453 bad packets, 800 incomplete data, 1300 time aberrations. These, in turn, could provide some profile of a particular camera helping to triage the panoply is issues. Maybe even have a facility for pushing the logs files to a server which digests the entries?

scottlamb commented 3 years ago

I've moved a little in the direction of log-but-continue with eg dadbd44 (which logs about something that used to be a fatal error, respecting enabled log levels), and I'll likely continue to as Retina gets more mature. But it's a fair bit of work to generate excellent logs, especially after things start going downhill. Eg once we're no longer sure we're at a correct packet boundary, every assumption we make from then on is suspect.

A central server is even more work, and I imagine many privacy-conscious users wouldn't be interested in using it, and/or we'd have to be very careful about what information it collects. The most useful thing to me is pcaps, followed by full TCP-level stream contents according to Retina. Either of those would be out of the question. Realistically, this would stay on my list for a very long time before I run out of tasks with lower cost and more certainty of pay-off.

scottlamb commented 3 years ago

For the Reolink RLC-420-5MP / hardware version IPC_51516M5M: at my request, Reolink built firmware v3.0.0.589_21091583 which advertises a=tool:LIVE555 Streaming Media v2020.08.12. It no longer has the stale TCP session bug, so there's no need to wait 65 seconds after an unclean shutdown to connect again.

The behavior when the TCP window fills is still bad. (I verified the current version of live555 is buggy; Reolink neither added nor fixed this bug.) But this is likely to come up less often now that there's no extra data from stale TCP sessions.

Reolink told me that the RLC-410 / hardware version IPC_3816M is too old and has been discontinued, so they won't be updating its firmware.

jlpoolen commented 3 years ago

For posterity, I purchased directly from Reolink 2 RLC-410 5mp cameras on 7/14/2021; $76.98 inc. shipping for 2 cameras. They arrived 7/24/21.

Here is the firmware they currently have Reolink_Client_2021-09-16_12-38-05 :

jlpoolen commented 3 years ago

I ran ffmpeg to see what the rtsp server version is. Note, you need to include "-loglevel debug " to get line 45 "a=tool:LIVE555 Streaming Media v2013.04.08"

jlpoole@taurus /usr/local/src $ ./ffmpeg1.sh 
Thu Sep 16 08:59:25 PM PDT 2021
Here is the log: /tmp/stderr_Sep_16_205901.log
 1  Commenced Thu Sep 16 08:59:01 PM PDT 2021
 2  Command:
 3     ffmpeg -loglevel debug -fflags nobuffer -i rtsp://moon:fire123@192.168.1.56:554/h264Preview_01_main -t 15 -y /tmp/Sep_16_205901.mp4 
 4     
 5  ffmpeg version 4.4 Copyright (c) 2000-2021 the FFmpeg developers
 6    built with gcc 10.3.0 (Gentoo 10.3.0-r2 p3)
 ...
39  [rtsp @ 0x55692ec55700] SDP:
40  v=0
41  o=- 1631548228552005 1 IN IP4 192.168.1.56
42  s=Session streamed by "preview"
43  i=h264Preview_01_main
44  t=0 0
45  a=tool:LIVE555 Streaming Media v2013.04.08
46  a=type:broadcast
47  a=control:*
48  a=range:npt=0-
49  a=x-qt-text-nam:Session streamed by "preview"
50  a=x-qt-text-inf:h264Preview_01_main
51  m=video 0 RTP/AVP 96
52  c=IN IP4 0.0.0.0
53  b=AS:500
54  a=rtpmap:96 H264/90000
55  a=fmtp:96 packetization-mode=1;profile-level-id=640033;sprop-parameter-sets=Z2QAM6zoAoAPGQ==,aO48sA==
56  a=control:trackID=1
57  m=audio 0 RTP/AVP 97
58  c=IN IP4 0.0.0.0
59  b=AS:256
60  a=rtpmap:97 MPEG4-GENERIC/16000
61  a=fmtp:97 streamtype=5;profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1408; profile=1;
62  a=control:trackID=2
63
64  Failed to parse interval end specification ''
65  [rtsp @ 0x55692ec55700] video codec set to: h264
66  [rtsp @ 0x55692ec55700] RTP Packetization Mode: 1

Conclusion: The change from 2013 to 2020 of the LIVE555 server in the Reolink camera occurred after July 2021.

scottlamb commented 2 years ago

I think I'm going to call this done. The major improvements here we've made to date are:

I haven't added any kind of "resync" to recover from these cameras' buggy behavior after the TCP window fills. Maybe I'll add that yet, but I'm not seeing complaints right now anyway.

hn commented 2 years ago

I would like to kindly point out my findings on Reolink cameras if this is not already known: Reolink RLC-410-5MP IP camera firmware unpacker and reverse engineered technical details. Maybe someone is able to patch the camera Linux system so that the Live555 server is updated (... or even replaced with a completely different streaming-server).

jlpoolen commented 2 years ago

Wow! I gave this a quick 2 minute scan and felt like I had entered Heaven. This is fantastic. I had contacted George Hillard and he did not reply so I gave up. But what you have accomplished here is what I have dreamt about if i had the time. I've been pursuing a Raspberyy Pi based approach using vetted hadware components and then hoping to use gstreamer and then whatever Scott comes up with that gives a no problem stream, or if there are problems, they can be identified as to what the cause is, i.e. long CATV lines causing voltage drops that is afflicting the hardware, or packet congestion at a node such as the PoE router.

I cannot wait to sink my teeth into your project, though I'm going to help the GenPI64 project if I can to create Gentoo images for Raspberry Pi so I have a completely controlled playing field.

As for LIVE555 -- I really do not think the maintainter's approach of limited transparency is good and possibly a red flag. I suspect he may have conflicted interests in having his code used by camera manufacturers such as Reolink and projects like VLC and therefore is unwilling to stage the codebase in a repository that makes the history of changes fully transparent. That plus his requirement that I must use an email address other than gmail to post to his list despite his knowing I've been a developer at a major software company for over 22 years.

I can't wait to carefully read your project. (I've been embarking on the side to learn electronics... again so I can look at something like the Reolink circuit board and have a better understanding of their design. I also confess I'm tired of my Zylex switches dying because of a cheap capacitor in the power board -- twice I hunted down the blown capacitors and replace them so I could resuce the switch. I have one now where my capacitor tests indicate all of the capacitors are okay, so I need to learn more to determine the power unit has failed and how to fix, or better yet, replace it. All this inspired by a posting on the Amazon comments for the Zylex showing a way on how to replace the blown capacitors.) I've digressed, but I do that when i get excited after finding projects such as yours and Scott's. I'll be in touch.

John

1566 Court ST NE Salem, OR 97301-4241 (707) 812-1323

On Thu, Mar 10, 2022, 1:58 AM Hajo Noerenberg @.***> wrote:

I would like to kindly point out my findings on Reolink cameras if this is not already known: Reolink RLC-410-5MP IP camera firmware unpacker and reverse engineered technical details https://github.com/hn/reolink-camera. Maybe someone is able to patch the camera Linux system so that the Live555 server is updated (... or even replaced with a completely different streaming-server).

— Reply to this email directly, view it on GitHub https://github.com/scottlamb/retina/issues/17#issuecomment-1063873218, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABXP4JK66B7LJYNILKTN6TU7HBSVANCNFSM5CFRZCNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>