Missing reference frames

[dewi-ny-je]

I'm recording video using RTSP H264 and codec "VBR 5000 kbps 15 fps". Sometimes, especially when the motion in the frame is not extensive, I get the whole 13 seconds of recording (even if I set 10 seconds...) without a single reference frame. This makes the video basically useless.

A reference frame should be introduced every second in the stream to ensure properly recorded videos.

[fundef1]

I believe the script use avconv to concatenate the images into a video. Ffmpeg is probably a better solution – if somebody could compile that…

I’ve use ffmpeg before to grab (MJPEG) streams directly from the RTSP stream. (mjpeg because it’s way more efficient as it avoids an entire encode decode cycle) But ffmpeg can also be use to concatenate images into a video.

From: dewi-ny-je notifications@github.com Sent: Tuesday, 12 May 2020 00:27 To: EliasKotlyar/Xiaomi-Dafang-Hacks Xiaomi-Dafang-Hacks@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [EliasKotlyar/Xiaomi-Dafang-Hacks] Missing reference frames (#1423)

I'm recording video using RTSP H264 and codec "VBR 5000 kbps 15 fps". Sometimes, especially when the motion in the frame is not extensive, I get the whole 13 seconds of recording (even if I set 10 seconds...) without a single reference frame. This makes the video basically useless.

A reference frame should be introduced every second in the stream to ensure properly recorded videos.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/EliasKotlyar/Xiaomi-Dafang-Hacks/issues/1423, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC5KXKFS426WQKYUIJLICDLRRB3SFANCNFSM4M6JXGZA.

[nanoamp]

If you have motion-triggered recording from the RTSP server then (providing you have set video_use_rtsp to true) it'll buffer the raw RTSP data directly, rather than using avconv.

So, the fix would likely be upstream with the v4l2rtspserver, persuading it to insert more frequent I-frames. The config for the server is setup in controlscripts/rtsp-h264 using data from config/rtspserver.conf. Looking upstream it's suggested that using the -c parameter for the rtspserver might help. Adding it to the RTSPH264OPTS in the conf file doesn't seem to make much difference on my cam though.

I'd be grateful if anyone else can come up with a way to improve this.

[sshaikh]

I can confirm that -c did nothing for me - the GOP/iframe/keyframe seems set to 84 regardless of other settings. So yeah, probably best to lobby on the upstream issue.

[sshaikh]

The author of upstream suggests another program, v4l2-ctl, can be used to control keyframes: https://github.com/mpromonet/v4l2rtspserver/issues/142#issuecomment-651394757

Is there a quick way to compile the binary for dafang?

[nik0]

@sshaikh You have to know that mpromonet/v4l2rtspserver is originaly for Raspberry. So the underlying API are different.

If someone want to play with the SDK, documentation: https://github.com/tssva/Ingenic-SDK-T10T20-3.12.0-20180320/tree/master/doc He is welcome to find to right API that controls the frequency of i-frame ...

[sshaikh]

Thanks @nik0 . But I see in another issue you have introduced MaxGop - isn't that the same thing?

https://github.com/EliasKotlyar/Xiaomi-Dafang-Hacks/issues/1488#issuecomment-642812448

[nik0]

it should be but I am afraid it is not so simple. Try to play with the parameters in the .ini file, and you tell me ...

[sshaikh]

I switched to the beta and: on the one hand MaxGop doesn't seem to do anything, you're right.

On the other hand, the keyframe frequency seems to be 25 which is a vast improvement on the 84 before. Would like to be able to reduce it further (to around 7) but hey, I'll take this for now.

[Dopeyr]

@sshaikh Out of interest, how are you measuring where the key frames are? I'd like to be able to view this somehow.

[sshaikh]

First you need to dump the rtsp stream:

ffmpeg -i rtsp://camera -vcodec copy -t 300 -y file.mp4

Then you need to analyse the dumped file:

ffprobe -show_frames -select_streams v:0 \
        -print_format csv file.mp4 2> /dev/null |
stdbuf -oL cut -d ',' -f4 |
grep -n 1 |
stdbuf -oL cut -d ':' -f1

This command will give you a list of key frame references, from that it's quite easy to see what the difference between counts are.

There may be a better way (eg to do it directly on the stream) but I'm dumping the streams already so I have files already to analyse. Credit goes to various stack overflow threads :)

EDIT: Possibly a better way:

ffprobe -select_streams v -show_frames -show_entries frame=pict_type,coded_picture_number frametest.mp4

[Dopeyr]

Thanks @sshaikh, that's really useful. I'll take a look.

[Dopeyr]

Hi, so I've had a quick play. It looks like maxgop setting does have an affect, but you can't set it lower than the framerate.

e.g. my framerate is 25 fps, and I set the maxgop to 100, so one key frame per 4 seconds. For a 30 second clip, I end up with:

94
194
294
394
494
594

For maxgop=1, I see

1
26
51
76
101
126
151
176
201
226
251
276
301
326
351
376
401
426
451
476
501
526
551
576
601

...but they're one every 25 frames (1 per second).

There are 25 key frames in total, but 25 != 30, so not quite sure what's going on there. Maybe time to buffer at the start or something?

[sshaikh]

That's interesting - I set my camera to 7fps and can't get the keyframes lower than every 25 frames. At the same time VLC thinks that my stream is running at 25fps when it's patently not.

I'll set it to 1 and see what happens.

[Dopeyr]

Hi, yeah that's something I noticed too - the framerate in VLC says 25fps, the framerate in the ffmpeg command above shows 25fps to start with, but then gradually heads towards 20fps. TinyCam also shows 20fps for me. I think the stream metadata is incorrect.

[sshaikh]

In the meantime I definitely can't get less that 25 per keyframe. At 7fps that means a new keyframe every 3 seconds or so which isn't ideal but better than when it was 85.

[Dopeyr]

Hi, so I've been doing a bit of digging. This is all new to me, so could be jumping to the wrong conclusions!

So according to wikipedia GOP (group of pictures) starts with a key frame (or intra frame).

I'm thinking that the "maxgop" setting is the "maximum number of frames in a GOP", because I've been experimenting with this setting, plus the bitrate setting.

If you set a really low bitrate, e.g. 100 and point the camera at a busy scene (I've been using the TV), you see a lot of artefacts, but at maxgop / fps seconds, you see the whole image "refresh", and is much better quality for a short time. I believe that is the keyframe.

e.g. If I set my fps to 25, and my maxgop to 125 I see the "refresh" (what I think is the keyframe) every 5 seconds. (125 / 25 = 5)

Using ffprobe, I see 4 keyframes listed for a 30s clip, which is roughly right.

If I set my maxgop to 50, I see the "refresh" every 2 seconds.

However, I think you're right with a maxgop setting less than 20-25-ish. If I set e.g. 10fps, my maxgop setting makes no difference until I get to 20+. At 10fps I see a keyframe every 2 seconds.

So it seems maxgop does control the number of keyframes, but you only see it with higher frame rates, due to some sort of hard limit where the max frames in a GOP cannot be less than 20-25ish.

i.e. for 7 fps, you'll see a keyframe every 20 / 7 = ~3 seconds, as you've observed.

[sshaikh]

Right. As you observed a few posts up, it works up to a limit. It could be memory or bandwidth capped, but then I would expect lower gops at lower resolutions.

In the meantime I've observed a similar situation with bitrate https://github.com/EliasKotlyar/Xiaomi-Dafang-Hacks/issues/1529.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dewi-ny-je]

@sshaikh and @Dopeyr so what is the step by step guide to reduce the i-frame interval to the framerate? you mention a beta? Can you maybe help @nik0 to patch the main branch so that everyone can get this (essential!) improvement?

As of now, having like security videos taken and not seeing anything for few seconds makes the camera useless.

[sshaikh]

There isn't one with the provided binaries. The best I've gotten is once every 25 frames at 7fps, ie every 3 seconds.

I suspect a correct solution relies on a code change upstream, but that's above my paygrade I'm afraid.

[dewi-ny-je]

Compared to the current stable release, one key frame every 25 is already a significant improvement. If I understood correctly, it's only for the beta branch? Can this improvement be brought to main branch? I see some commits from May-July related to maxgop and to an updated vl42rtspserver.

I'm asking because I'mnot sure how the beta branch is handled, since I see many commits from several months ago which haven't made it in the main branch.

[sshaikh]

So, apparently the alternative FW at https://github.com/openmiko/openmiko correctly handles the GOP parameter, so if that's something vital to control that's an option. There's no UI though so you'll have to pick what's important!

[dewi-ny-je]

It was mentioned above that the v4l2rtspserver in the beta branch is improved over the main branch and in fact I checked and the beta has a new parameter to control the GOP.

I could send a pull request to merge the two, I have tried it at home and the merge is not automatic but nevertheless trivial. Also, it seems that things work properly.

I'll test it for few days and if I don't find any obvious bug I'll send the request.

It feels wrong to be the one doing it, since I did not contribute to the beta but only to the main branch, however it seems the beta has not been touched for a while.

[sshaikh]

Well yeh, we're the ones who mentioned that :). There is a GOP parameter but it doesn't work fully as intended. If that is sufficient for you then that's great.

But a GOP of 25 might be too much for low FPS users like myself. With the alternative FW you can get as low as a GOP of 5 in 25fps (ie every 1/5 of a second).

[dewi-ny-je]

I don't need a GUI, but I also don't want to step into newly born projects. In openmiko the developer stated that motion detection is not implemented yet. That's enough to put it on hold for me, but I starred it for future reminder.

I can switch next year... Now I try to make this project better by making the beta branch available to everyone!

I use 15-25 fps so gop=25 is fine to me.

[dewi-ny-je]

I'm using the beta. Standard settings, CBR, 5000 kbps, FullHD, 25 fps. The recorded video is marked by VLC as 15 fps, which is ok anyway, and the effective bitrate is 5800 kbps, also no problem.

Within 2 seconds I get a reference frame, which is almost perfect! surely enough for the purpose.

I'll proceed to request merging of beta and master, this and all the other improvements seem to work fine and to be useful.

[jmtatsch]

Master and beta have diverged quite a bit. If you can reconcile them I would happily merge that.