search

scottlamb / moonfire-nvr

Moonfire NVR, a security camera network video recorder
Other
1.18k stars 138 forks source link

Camera feed full of disconnects and short segments #302

Closed a-bali closed 1 month ago

a-bali commented 5 months ago

I'm trying to run Moonfire NVR with a cheap Chinese wifi security camera (not sure if it has any meaningful brand or version, comes with the ICSee mobile app). The camera which works fine with ffmpeg via TCP/RTSP, producing continuous 1-hour segments without problems with the following command: ffmpeg -rtsp_transport tcp -i rtsp://ip:554/user=..._password=..._channel=1_stream=0.sdp?real_stream -an -vcodec copy -f segment -segment_time 600 -strftime 1 -loglevel warning ...

With latest (0.7.12) Moonfire NVR (same URL as above, TCP transport) however, the feed is full of random disconnects (sometimes after just a few seconds, sometimes after minutes, but sometimes only after a couple of hours - see screenshot below), therefore the list of segments is very fragmented and of course it is also not possible to watch an arbitrary timespan at once with these small fragments.

Would it be possible to somehow improve the feed continuity as ffmpeg does (I don't get any disconnects there) or merge the segments in the background and thus hide the disconnects from the user?

Let me know if I can somehow provide further data for debugging.

image

scottlamb commented 5 months ago

Could you attach some logs? They should explain each disconnection.

a-bali commented 5 months ago

Below is an example for a disconnect:

024-01-14T08:34:50.442156 DEBUG tokio-runtime-worker streamer{stream="nyaralo-main"}: retina::client: RTSP connection error: RtspFramingError { conn_ctx: ConnectionContext { local_addr: 192.168.33.100:46922, peer_addr: 192.168.33.200:554, established_wall: WallTime(Timespec { sec: 1705217450, nsec: 319740990 }) }, msg_ctx: RtspMessageContext { pos: 19897610, received_wall: WallTime(Timespec { sec: 1705217690, nsec: 442137263 }), received: Instant { tv_sec: 1381958, tv_nsec: 928713485 } }, description: "Invalid RTSP message; buffered:\nLength: 1462 (0x5b6) bytes\n0000:   e2 12 cb ab  b0 e8 5e 22  fc 3b 2d ce  0c 9d cc 26   ......^\".;-....&\n0010:   ae 7f a7 77  89 f5 89 41  ff 39 e1 a3  a6 3f 59 7e   ...w...A.9...?Y~\n0020:   aa 97 b1 26  76 ed bb 40  cd 67 e3 81  7f db 2b 03   ...&v..@.g....+.\n0030:   5b 75 24 00  05 80 80 60  37 c1 b7 7b  b4 3f 00 00   [u$....`7..{.?..\n0040:   00 00 3c 05  48 41 31 e9  f3 92 82 ce  62 6d af b9   ..<.HA1.....bm..\n0050:   e2 e9 51 00  e2 41 29 50  f9 d1 ee 24  70 40 da 31   ..Q..A)P...$p@.1\n0060:   84 cf 92 4e  84 65 ef df  ff df f2 26  f5 0e d2 e4   ...N.e.....&....\n0070:   55 e5 1a 1d  85 69 2f ab  92 87 4c aa  3f b3 82 f5   U....i/...L.?...\n...1334 (0x536) bytes not shown..." }
2024-01-14T08:34:50.443530 DEBUG tokio-runtime-worker streamer{stream="nyaralo-main"}: retina::client: "nyaralo"/2 tracking TEARDOWN of session id 416734350
2024-01-14T08:34:50.445384  WARN       s-nyaralo-main streamer{stream="nyaralo-main"}: moonfire_nvr::streamer: sleeping for 1 s after error err=UNKNOWN
caused by: RTSP framing error: Invalid RTSP message; buffered:
Length: 1462 (0x5b6) bytes
0000:   e2 12 cb ab  b0 e8 5e 22  fc 3b 2d ce  0c 9d cc 26   ......^".;-....&
0010:   ae 7f a7 77  89 f5 89 41  ff 39 e1 a3  a6 3f 59 7e   ...w...A.9...?Y~
0020:   aa 97 b1 26  76 ed bb 40  cd 67 e3 81  7f db 2b 03   ...&v..@.g....+.
0030:   5b 75 24 00  05 80 80 60  37 c1 b7 7b  b4 3f 00 00   [u$....`7..{.?..
0040:   00 00 3c 05  48 41 31 e9  f3 92 82 ce  62 6d af b9   ..<.HA1.....bm..
0050:   e2 e9 51 00  e2 41 29 50  f9 d1 ee 24  70 40 da 31   ..Q..A)P...$p@.1
0060:   84 cf 92 4e  84 65 ef df  ff df f2 26  f5 0e d2 e4   ...N.e.....&....
0070:   55 e5 1a 1d  85 69 2f ab  92 87 4c aa  3f b3 82 f5   U....i/...L.?...
...1334 (0x536) bytes not shown...

conn: 192.168.33.100:46922(me)->192.168.33.200:554@2024-01-14T08:30:50
msg: 19897610@2024-01-14T08:34:50
2024-01-14T08:34:50.446312 DEBUG tokio-runtime-worker retina::client::teardown: TEARDOWN 416734350 starting for URL rtsp://192.168.33.200:554/user=XXX_password=XXX_channel=1_stream=0.sdp/
2024-01-14T08:34:50.447888 DEBUG tokio-runtime-worker retina::client::teardown: TEARDOWN 416734350 on existing conn failed: Error reading from RTSP peer: EOF while expecting response to TEARDOWN CSeq 12

conn: 192.168.33.100:46922(me)->192.168.33.200:554@2024-01-14T08:30:50
msg: 19899072@2024-01-14T08:34:50
2024-01-14T08:34:50.449692 DEBUG tokio-runtime-worker retina::client::teardown: Giving up on TEARDOWN 416734350; use TearDownPolicy::Always to try harder
2024-01-14T08:34:50.453294 DEBUG               sync-1 syncer{path=/home/bali/nvr/main}: moonfire_db::writer: 1: have remaining quota of -13546938368
2024-01-14T08:34:50.463051  INFO               sync-1 syncer{path=/home/bali/nvr/main}:flush{flush_count=15 reason="0 sec after start of 39 seconds nyaralo-main recording 1/2440"}: moonfire_db::db: flush complete:
/home/bali/nvr/main: added 2M 830K 434B in 1 recordings (1/2440), deleted 0B in 0 (), GCed 0 recordings ().
2024-01-14T08:34:51.446359  INFO       s-nyaralo-main streamer{stream="nyaralo-main"}: moonfire_nvr::streamer: opening input url=rtsp://192.168.33.200:554/user=XXX_password=XXX_channel=1_stream=0.sdp?real_stream

    seq_parameter_set_id: ParamSetId(
        0,
    ),
    chroma_info: ChromaInfo {
        chroma_format: YUV420,
        separate_colour_plane_flag: false,
        bit_depth_luma_minus8: 0,
        bit_depth_chroma_minus8: 0,
        qpprime_y_zero_transform_bypass_flag: false,
        scaling_matrix: SeqScalingMatrix,
    },
    log2_max_frame_num_minus4: 12,
    pic_order_cnt: TypeTwo,
    max_num_ref_frames: 2,
    gaps_in_frame_num_value_allowed_flag: true,
    pic_width_in_mbs_minus1: 119,
    pic_height_in_map_units_minus1: 67,
    frame_mbs_flags: Frames,
    direct_8x8_inference_flag: true,
    frame_cropping: Some(
        FrameCropping {
            left_offset: 0,
            right_offset: 0,
            top_offset: 0,
            bottom_offset: 4,
        },
    ),
    vui_parameters: Some(
        VuiParameters {
            aspect_ratio_info: None,
            overscan_appropriate: Unspecified,
            video_signal_type: None,
            chroma_loc_info: None,
            timing_info: Some(
                TimingInfo {
                    num_units_in_tick: 1,
                    time_scale: 24,
                    fixed_frame_rate_flag: true,
                },
            ),
            nal_hrd_parameters: None,
            vcl_hrd_parameters: None,
            low_delay_hrd_flag: None,
            pic_struct_present_flag: false,
            bitstream_restrictions: None,
        },
    ),
}
2024-01-14T08:34:51.457859 DEBUG tokio-runtime-worker streamer{stream="nyaralo-main"}: moonfire_nvr::stream: connected to "nyaralo-main", tool None
2024-01-14T08:34:51.464620 DEBUG tokio-runtime-worker streamer{stream="nyaralo-main"}: retina::client: SETUP response: Response {
    version: V1_0,
    status: Ok,
    reason_phrase: "OK",
    headers: Headers(
        {
            HeaderName(
                "Cache-Control",
            ): HeaderValue(
                "private",
            ),
            HeaderName(
                "Cseq",
            ): HeaderValue(
                "2",
            ),
            HeaderName(
                "Server",
            ): HeaderValue(
                "H264DVR 1.0",
            ),
            HeaderName(
                "Session",
            ): HeaderValue(
                "416975520;timeout=60",
            ),
            HeaderName(
                "Transport",
            ): HeaderValue(
                "RTP/AVP/TCP;unicast;interleaved=0-1;mode=PLAY",
            ),
            HeaderName(
                "x-Dynamic-Rate",
            ): HeaderValue(
                "1",
            ),
        },
    ),
    body: b"",
}
2024-01-14T08:34:51.466364 DEBUG tokio-runtime-worker streamer{stream="nyaralo-main"}: retina::client: established session "416975520", timeout=60s
2024-01-14T08:34:51.488023 DEBUG tokio-runtime-worker streamer{stream="nyaralo-main"}: retina::client: no PLAY seq on stream 0
2024-01-14T08:34:51.643325 DEBUG       s-nyaralo-main streamer{stream="nyaralo-main"}: moonfire_nvr::streamer: have first key frame
2024-01-14T08:34:51.646864  WARN tokio-runtime-worker streamer{stream="nyaralo-main"}: retina::client: Ignoring data on unassigned RTSP interleaved data channel 2. This is the first such message. Following messages will be logged at trace priority only.

conn: 192.168.33.100:47604(me)->192.168.33.200:554@2024-01-14T08:34:51
msg: 330606@2024-01-14T08:34:51
data: Length: 172 (0xac) bytes
0000:   80 88 00 00  c6 63 70 60  00 00 00 00  55 55 54 54   .....cp`....UUTT
0010:   54 56 56 56  56 51 50 50  50 50 50 53  53 52 52 5d   TVVVVQPPPPPSSRR]
0020:   5c 5c 5d 52  52 5d 5c 5c  5c 5c 5d 52  52 52 5c 59   \\]RR]\\\\]RRR\Y
0030:   5a 44 47 41  43 42 4d 4d  4d 4f 49 4e  49 49 49 48   ZDGACBMMMOINIIIH
0040:   48 4e 4f 4e  4e 4f 4f 4c  4d 42 40 46  44 45 5a 58   HNONNOOLMB@FDEZX
0050:   59 5f 5c 5c  5c 52 53 52  53 50 51 51  56 57 57 57   Y_\\\RSRSPQQVWWW
0060:   55 d5 d4 d7  d4 d4 d7 d6  d0 d0 d0 d1  d6 d6 d1 d1   U...............
0070:   d0 d0 d1 d1  d1 d0 d0 d0  d0 d1 d6 d6  d1 d6 d1 d2   ................
...44 (0x2c) bytes not shown...
2024-01-14T08:36:01.647113 DEBUG               sync-1 syncer{path=/home/bali/nvr/main}: moonfire_db::writer: 1: have remaining quota of -13541707776
2024-01-14T08:36:01.658357  INFO               sync-1 syncer{path=/home/bali/nvr/main}:flush{flush_count=16 reason="0 sec after start of 1 minute 10 seconds nyaralo-main recording 1/2441"}: moonfire_db::db: flush complete:
a-bali commented 5 months ago

Also these "parameter changes":

2024-01-15T06:15:34.885377 DEBUG tokio-runtime-worker streamer{stream="nyaralo-main"}: retina::codec::h264: sps: SeqParameterSet {
    profile_idc: ProfileIdc(
        77,
    ),
    constraint_flags: ConstraintFlags {
        flag0: false,
        flag1: false,
        flag2: false,
        flag3: false,
        flag4: false,
        flag5: false,
        reserved_zero_two_bits: 0,
    },
    level_idc: 41,
    seq_parameter_set_id: ParamSetId(
        0,
    ),
    chroma_info: ChromaInfo {
        chroma_format: YUV420,
        separate_colour_plane_flag: false,
        bit_depth_luma_minus8: 0,
        bit_depth_chroma_minus8: 0,
        qpprime_y_zero_transform_bypass_flag: false,
        scaling_matrix: SeqScalingMatrix,
    },
    log2_max_frame_num_minus4: 12,
    pic_order_cnt: TypeTwo,
    max_num_ref_frames: 2,
    gaps_in_frame_num_value_allowed_flag: true,
    pic_width_in_mbs_minus1: 119,
    pic_height_in_map_units_minus1: 67,
    frame_mbs_flags: Frames,
    direct_8x8_inference_flag: true,
    frame_cropping: Some(
        FrameCropping {
            left_offset: 0,
            right_offset: 0,
            top_offset: 0,
            bottom_offset: 4,
        },
    ),
    vui_parameters: Some(
        VuiParameters {
            aspect_ratio_info: None,
            overscan_appropriate: Unspecified,                                                                                                                                                                      video_signal_type: None,
            chroma_loc_info: None,
            timing_info: Some(
                TimingInfo {
                    num_units_in_tick: 1,
                    time_scale: 24,
                    fixed_frame_rate_flag: true,
                },
            ),
            nal_hrd_parameters: None,
            vcl_hrd_parameters: None,
            low_delay_hrd_flag: None,
            pic_struct_present_flag: false,
            bitstream_restrictions: None,
        },
    ),
}
2024-01-15T06:15:34.886011 DEBUG       s-nyaralo-main streamer{stream="nyaralo-main"}: moonfire_nvr::stream: nyaralo-main: parameter change:
old: VideoSampleEntryToInsert { data: Length: 131 (0x83) bytes
0000:   00 00 00 83  61 76 63 31  00 00 00 00  00 00 00 01   ....avc1........
0010:   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00   ................
0020:   07 80 04 38  00 48 00 00  00 48 00 00  00 00 00 00   ...8.H...H......
0030:   00 01 00 00  00 00 00 00  00 00 00 00  00 00 00 00   ................
0040:   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00   ................
0050:   00 00 00 18  ff ff 00 00  00 2d 61 76  63 43 01 4d   .........-avcC.M
0060:   00 29 ff e1  00 15 27 4d  00 29 8d 6e  07 80 22 7e   .)....'M.).n.."~
0070:   58 40 00 00  03 00 40 00  00 06 21 01  00 05 28 ee   X@....@...!...(.
0080:   02 1c 80                                             ..., rfc6381_codec: "avc1.4d0029", width: 1920, height: 1080, pasp_h_spacing: 1, pasp_v_spacing: 1 }
new: VideoSampleEntryToInsert { data: Length: 130 (0x82) bytes
0000:   00 00 00 82  61 76 63 31  00 00 00 00  00 00 00 01   ....avc1........
0010:   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00   ................
0020:   07 80 04 38  00 48 00 00  00 48 00 00  00 00 00 00   ...8.H...H......
0030:   00 01 00 00  00 00 00 00  00 00 00 00  00 00 00 00   ................
0040:   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00   ................
0050:   00 00 00 18  ff ff 00 00  00 2c 61 76  63 43 01 4d   .........,avcC.M
0060:   00 29 ff e1  00 15 27 4d  00 29 8d 6e  07 80 22 7e   .)....'M.).n.."~
0070:   58 40 00 00  03 00 40 00  00 06 21 01  00 04 28 ee   X@....@...!...(.
0080:   05 72                                                .r, rfc6381_codec: "avc1.4d0029", width: 1920, height: 1080, pasp_h_spacing: 1, pasp_v_spacing: 1 }
2024-01-15T06:15:34.891664 DEBUG               sync-1 syncer{path=/home/bali/nvr/main}: moonfire_db::writer: 1: have remaining quota of -10882781184
2024-01-15T06:15:34.897675  INFO               sync-1 syncer{path=/home/bali/nvr/main}:flush{flush_count=1580 reason="0 sec after start of 19 seconds nyaralo-main recording 1/4005"}: moonfire_db::db:
flush complete:
/home/bali/nvr/main: added 195K 163B in 1 recordings (1/4005), deleted 0B in 0 (), GCed 0 recordings ().
scottlamb commented 5 months ago

caused by: RTSP framing error: Invalid RTSP message; buffered:

This one is unfortunate. I've seen something similar with cameras using the live555 library. When the TCP window fills (maybe as a result of wifi congestion or something), they get a partial write—the kernel wrote what it could to the buffer and returned to userspace. They don't handle this well. The correct thing to do would be to monitor for write availability, then when the kernel indicates that, repeat the write with only the remaining portion. But poorly written cameras do all manner of dumb things instead—repeating the whole write, skipping the remaining bytes, blocking the whole thread (that's supposed to be doing other things) until there's availability, dropping the connection right away. The first two break the protocol, producing this error.

If you got a packet capture with e.g. Wireshark it'd probably confirm that the TCP window filled shortly before this happened.

So why does ffmpeg keep the connection? Basically I think they skip forward in the incoming stream until they find something that looks like a properly framed message. The preceding message is likely corrupt; and some stuff gets skipped; but they keep the connection open where Retina/Moonfire currently drops it. I'm not thrilled about that approach but it is possible to support doing the same in Retina. 🤷

You could try using UDP. Some stuff may still get dropped (because RTP-over-UDP doesn't have any retry logic at all) but at least there wouldn't be any corrupt RTP messages. There's still an RTSP TCP connection that in theory can be affected by the same bug, but it's only used for keepalives after the session is started, so it's unlikely that the TCP window will ever fill.

Also these "parameter changes":

I'm not sure what the actual change was here; it must have been pretty subtle. I'll dig into those bytes later. But it's probably not too hard to make these ones appear as one entry and play without break.

Would it be possible to ... merge the segments in the background and thus hide the disconnects from the user?

fwiw, my ideal UI would be totally different, based on a scrub bar sort of thing. You select when to start playing, and it would likely jump to the next segment any time the connection was lost like this. But it's a lot of effort to implement and I don't know when I'll find the time for it... :-(

scottlamb commented 5 months ago

The parameter change is the following...

--- before.debug    2024-01-15 13:40:20
+++ after.debug 2024-01-15 13:40:26
@@ -78,7 +78,7 @@
             num_ref_idx_l1_default_active_minus1: 0,
             weighted_pred_flag: false,
             weighted_bipred_idc: 0,
-            pic_init_qp_minus26: 8,
+            pic_init_qp_minus26: 5,
             pic_init_qs_minus26: 0,
             chroma_qp_index_offset: 0,
             deblocking_filter_control_present_flag: true,

...which is a detail of how the frames are encoded. I've never seen a camera change this sort of parameter mid-stream on its own, but it seems like a reasonable thing to do, and I don't think it will be a big change for Moonfire to make it seamless.

a-bali commented 5 months ago

Thanks for the detailed investigation!

You could try using UDP. Some stuff may still get dropped (because RTP-over-UDP doesn't have any retry logic at all) but at least there wouldn't be any corrupt RTP messages. There's still an RTSP TCP connection that in theory can be affected by the same bug, but it's only used for keepalives after the session is started, so it's unlikely that the TCP window will ever fill.

Strange thing is that the camera works with UDP using ffmpeg (with -rtsp_transport udp), but not with Moonfire, see below. I'm not sure I understand the client/server ports in the Retina client setup, maybe this could be the reason for getting a connection refused error?

2024-01-16T08:32:06.409783  INFO s-nyaralo-main streamer{stream="nyaralo-main"}: moonfire_nvr::streamer: opening input url=rtsp://192.168.33.200:554/user=XXX_password=XXX_channel=1_stream=0.sdp?real_stream
2024-01-16T08:32:06.421627 DEBUG tokio-runtime-worker streamer{stream="nyaralo-main"}: retina::codec::h264: sps: SeqParameterSet {
    profile_idc: ProfileIdc(
        77,
    ),
    constraint_flags: ConstraintFlags {
        flag0: false,
        flag1: false,
        flag2: false,
        flag3: false,
        flag4: false,
        flag5: false,
        reserved_zero_two_bits: 0,
    },
    level_idc: 41,
    seq_parameter_set_id: ParamSetId(
        0,
    ),
    chroma_info: ChromaInfo {
        chroma_format: YUV420,
        separate_colour_plane_flag: false,
        bit_depth_luma_minus8: 0,
        bit_depth_chroma_minus8: 0,
        qpprime_y_zero_transform_bypass_flag: false,
        scaling_matrix: SeqScalingMatrix,
    },
    log2_max_frame_num_minus4: 12,
    pic_order_cnt: TypeTwo,
    max_num_ref_frames: 2,
    gaps_in_frame_num_value_allowed_flag: true,
    pic_width_in_mbs_minus1: 119,
    pic_height_in_map_units_minus1: 67,
    frame_mbs_flags: Frames,
    direct_8x8_inference_flag: true,
    frame_cropping: Some(
        FrameCropping {
            left_offset: 0,
            right_offset: 0,
            top_offset: 0,
            bottom_offset: 4,
        },
    ),
    vui_parameters: Some(
        VuiParameters {
            aspect_ratio_info: None,
            overscan_appropriate: Unspecified,
            video_signal_type: None,
            chroma_loc_info: None,
            timing_info: Some(
                TimingInfo {
                    num_units_in_tick: 1,
                    time_scale: 24,
                    fixed_frame_rate_flag: true,
                },
            ),
            nal_hrd_parameters: None,
            vcl_hrd_parameters: None,
            low_delay_hrd_flag: None,
            pic_struct_present_flag: false,
            bitstream_restrictions: None,
        },
    ),
}
2024-01-16T08:32:06.422046 DEBUG tokio-runtime-worker streamer{stream="nyaralo-main"}: moonfire_nvr::stream: connected to "nyaralo-main", tool None
2024-01-16T08:32:06.427634 DEBUG tokio-runtime-worker streamer{stream="nyaralo-main"}: retina::client: SETUP response: Response {
    version: V1_0,
    status: Ok,
    reason_phrase: "OK",
    headers: Headers(
        {
            HeaderName(
                "Cache-Control",
            ): HeaderValue(
                "private",
            ),
            HeaderName(
                "Cseq",
            ): HeaderValue(
                "2",
            ),
            HeaderName(
                "Server",
            ): HeaderValue(
                "H264DVR 1.0",
            ),
            HeaderName(
                "Session",
            ): HeaderValue(
                "18426930;timeout=60",
            ),
            HeaderName(
                "Transport",
            ): HeaderValue(
                "RTP/AVP;unicast;mode=PLAY;source=192.168.33.200;client_port=20566-20567;server_port=40004-40005;ssrc=0",
            ),
            HeaderName(
                "x-Dynamic-Rate",
            ): HeaderValue(
                "1",
            ),
        },
    ),
    body: b"",
}
2024-01-16T08:32:06.427782 DEBUG tokio-runtime-worker streamer{stream="nyaralo-main"}: retina::client: established session "18426930", timeout=60s
2024-01-16T08:32:06.474171 DEBUG tokio-runtime-worker streamer{stream="nyaralo-main"}: retina::client: no PLAY seq on stream 0
2024-01-16T08:32:06.560364 DEBUG tokio-runtime-worker streamer{stream="nyaralo-main"}: retina::client: Ignoring UDP connection refused error
scottlamb commented 4 months ago

I'll roll a new release soon-ish (hopefully tonight) with the commit above that should seamlessly handle those parameter change events, rather than causing separate rows in the UI. I don't have a camera that does this though so I'll need your feedback on if it's working as well as I hope!

As for the UDP stuff, the "connection refused" stuff should be totally unimportant. (When I get around to it, I'll alter the message to make this clearer.) Is there any following log message that might explain what's going on? I might need a packet capture (e.g. with Wireshark) to understand this if not.

scottlamb commented 4 months ago

I don't have a camera that does this though so I'll need your feedback on if it's working as well as I hope!

Hmm, actually, on one of my cameras, if I change the frame rate in the UI, it alters the video sample entry without ending the stream. This isn't frequent/unprompted as with your camera, but it allows me to test my change. Seems to work!

scottlamb commented 4 months ago

I'll roll a new release soon-ish (hopefully tonight) with the commit above that should seamlessly handle those parameter change events, rather than causing separate rows in the UI.

https://github.com/scottlamb/moonfire-nvr/releases/tag/v0.7.13

scottlamb commented 2 months ago

Did this have a significant effect? The remaining short streams should be actual disconnects. I also have been meaning to add a mouseover to the UI that shows the reason for a disconnect rather than requiring you go to the logs.