seamless mid-stream video parameter changes

scottlamb commented 2 years ago

In https://github.com/scottlamb/moonfire-nvr/issues/213#issuecomment-1094110481, @IronOxidizer wrote:

For the most part [my camera with Retina's mp4 example program] seems to be working, though after a few minutes (or even seconds, it's inconsistent) I get the following error. Seems like an issue parsing a keyframe since I only get it after a large byte video frame.

I20220409 15:15:12.382 main client::mp4] Using h264 video stream
I20220409 15:15:12.389 main client::mp4] Ignoring pcma audio stream because it c                                                                                                                                                             an't be placed into a .mp4 file without transcoding
I20220409 15:15:12.389 main client::mp4] No suitable audio stream found
W20220409 15:15:12.414 main retina::client] Ignoring data on unassigned RTSP int                                                                                                                                                             erleaved data channel 2. This is the first such message. Following messages will                                                                                                                                                              be logged at trace priority only.

conn: 192.168.1.64:33994(me)->192.168.1.71:554@2022-04-09T15:15:12
msg: 954@2022-04-09T15:15:12
data: Length: 172 (0xac) bytes
0000:   80 88 00 00  1a 91 98 f0  00 00 00 01  d4 d4 d4 d4   ................
0010:   d4 d7 d7 d7  d7 d4 d4 d4  d5 d5 d5 d5  d5 d5 d5 d5   ................
0020:   d5 d5 d4 d4  d4 d4 d4 d4  d5 d5 d4 d4  d4 d7 d7 d7   ................
0030:   d7 d4 d4 d5  d5 d5 55 55  55 55 d5 55  55 d5 d5 d5   ......UUUU.UU...
0040:   d4 d5 d4 d4  d4 d4 d4 d4  d4 d7 d4 d7  d4 d5 d4 d5   ................
0050:   d5 d4 d4 d4  d4 d4 d4 d4  d5 d4 d4 d5  d4 d4 d4 d4   ................
0060:   d5 d5 d5 d5  d5 d4 d4 d7  d7 d7 d7 d4  d4 d4 d5 d5   ................
0070:   d5 d5 d5 d5  d5 d5 d5 d5  d5 d4 d4 d4  d4 d4 d5 d4   ................

...44 (0x2c) bytes not shown...
526865204 (mod-2^32: 526865204), npt 0.000: 250698-byte video frame
526872704 (mod-2^32: 526872704), npt 0.083: 1606-byte video frame
526880204 (mod-2^32: 526880204), npt 0.167: 1888-byte video frame
526887704 (mod-2^32: 526887704), npt 0.250: 1675-byte video frame
526895204 (mod-2^32: 526895204), npt 0.333: 2272-byte video frame
526902704 (mod-2^32: 526902704), npt 0.417: 606-byte video frame
526910204 (mod-2^32: 526910204), npt 0.500: 2057-byte video frame
526917704 (mod-2^32: 526917704), npt 0.583: 1872-byte video frame
526925204 (mod-2^32: 526925204), npt 0.667: 1787-byte video frame
526932704 (mod-2^32: 526932704), npt 0.750: 2730-byte video frame
526940204 (mod-2^32: 526940204), npt 0.833: 4246-byte video frame
526947704 (mod-2^32: 526947704), npt 0.917: 1768-byte video frame
526955204 (mod-2^32: 526955204), npt 1.000: 316962-byte video frame
E20220409 15:15:13.815 main client] Fatal: Error processing video frame starting                                                                                                                                                              with 287984@2022-04-09T15:15:13 ch=0
caused by: video parameters change unimplemented.
old: VideoParameters {
    rfc6381_codec: "avc1.4D0029",
    pixel_dimensions: (
        1920,
        1080,
    ),
    pixel_aspect_ratio: None,
    frame_rate: Some(
        (
            2,
            24,
        ),
    ),
    extra_data: Length: 37 (0x25) bytes
    0000:   01 4d 00 29  ff e1 00 15  27 4d 00 29  8d 6a 07 80   .M.)....'M.).j.                                                                                                                                                             .
    0010:   22 7e 58 40  00 00 03 00  40 00 00 06  21 01 00 05   "~X@....@...!..                                                                                                                                                             .
    0020:   28 ee 03 9c  80                                      (....
    ,
}
new: VideoParameters {
    rfc6381_codec: "avc1.4D0029",
    pixel_dimensions: (
        1920,
        1080,
    ),
    pixel_aspect_ratio: None,
    frame_rate: Some(
        (
            2,
            24,
        ),
    ),
    extra_data: Length: 36 (0x24) bytes
    0000:   01 4d 00 29  ff e1 00 15  27 4d 00 29  8d 6a 07 80   .M.)....'M.).j.                                                                                                                                                             .
    0010:   22 7e 58 40  00 00 03 00  40 00 00 06  21 01 00 04   "~X@....@...!..                                                                                                                                                             .
    0020:   28 ee 05 72                                          (..r
    ,
}

Out of curiosity, I parsed the extra_data above to see what changed:

--- before.debug    2022-04-09 19:39:25.796601500 -0700
+++ after.debug 2022-04-09 19:39:33.984746977 -0700
@@ -112,7 +112,7 @@
                 num_ref_idx_l1_default_active_minus1: 0,
                 weighted_pred_flag: false,
                 weighted_bipred_idc: 0,
-                pic_init_qp_minus26: 14,
+                pic_init_qp_minus26: 5,
                 pic_init_qs_minus26: 0,
                 chroma_qp_index_offset: 0,
                 deblocking_filter_control_present_flag: true,

The H.264 spec says:

slice_qp_delta specifies the initial value of QP_Y to be used for all the macroblocks in the slice until modified by the value of mb_qp_delta in the macroblock layer. The initial QP_Y quantisation parameter for the slice is computed as

SliceQP_Y = 26 + pic_init_qp_minus26 + slice_qp_delta

What QP_Y actually does is a little beyond my understanding, so let's just say it changes the image somehow.

Basically the camera is (re)setting a default parameter that may be used or always overridden is used in each macroblock (16x16 pixel block of an image). Of course, altering our code to inspect all the macroblocks to know which before deciding how to handle the parameter change would be madness. The only reasonable assumption is that the parameter change is important.

[edit: actually, even if QP_Y is later adjusted, the new value is written as a delta from the current. So the parameter change really does affect all following frames no matter what.]

Cameras are allowed to change video parameters mid-stream, although this is the first time I've seen one decide to on its own. Some of my cameras will do a mid-stream parameter change only if you actually change the image settings in the camera's config UI. The rest won't even then—they'll drop all active RTSP streams instead, forcing a reconnect. Moonfire's current approach is to reconnect on a mid-stream parameter change, basically treating the former like the latter. It's been good enough so far. But sounds like this camera might have extremely frequent drops if we don't rethink this approach.

The right way to handle this is to, at the .mp4 layer, plumb through the new parameters into a fresh "sample description" (terminology from ISO 14496-12). The Retina library is fine—it's already passing along that the parameters changed. The Retina mp4 example needs to change here. Moonfire's recording logic needs to change here to seamlessly change parameters without reconnecting, which will require some thought to plumb through properly to the layer above. And on playback, the .mp4 generation logic might already handle this fine but I haven't really tested this case. Likewise the live stream logic in the UI layer.

I think most players would let us get away without doing it the right way. Moonfire currently isn't stripping the video parameters out of the actual video frames (see #43). I think a lot of video players basically concatenate the sample descriptions together with the video frames, so basically the H.264 decoder won't even notice the wrong sample descriptions because they're immediately overridden. In other words, if we commented out that bail! line in Moonfire code so it just didn't pay attention to this parameter change (that notably doesn't affect anything like image resolution that Moonfire pays particular attention to), this would work fine for your camera. Probably.

I'd like to handle this the right way. Not sure yet how quickly it can happen. I've got some other things going on like finishing my taxes next week.

scottlamb commented 2 years ago

If you pull latest Retina, the mp4 example should no longer produce this error.

scottlamb commented 2 years ago

Progress. I updated Moonfire's recording logic. Wasn't as bad as I feared. It no longer disconnects and reconnects when the parameters change, instead creating a new recording within the run. That's assuming the parameters change at a key frame. If they change somewhere else...well, then it'll be a bit more involved.

Currently the UI's list view will display a new row in the table any time the parameters change, and it won't let you create a single .mp4 file that spans the change (even though the /view.mp4 endpoint in the API should handle that fine). This is basically because it displays the pixel dimensions for each row. We could alter it to only split up the run into multiple rows when the dimensions actually change. A future scrub bar UI (#32) also should handle this seamlessly.

The live video will break on parameter change right now. Here's the offending code:

https://github.com/scottlamb/moonfire-nvr/blob/3bc552b95004d13223120d1c3fcf5d43a6af83f6/ui/src/Live/LiveCamera.tsx#L215

scottlamb commented 2 years ago

The recording change is in the freshly released v0.7.4.

scottlamb commented 1 month ago

https://github.com/scottlamb/moonfire-nvr/commit/1f7c4c184a77db01cb0dfdec589bf9dd297ae0a4

scottlamb / moonfire-nvr

seamless mid-stream video parameter changes #217