Dynamic bitrate change (H.264)

[andrewvoznytsa]

As far as I see imxvpuenc_h264 is unable to change bitrate after switching to playing state. According to the source code it should work but instead of that in response to g_object_set(G_OBJECT(videnc), "bitrate", 2000, nullptr); I get the following:

0:00:54.106337301 21964 0xaaaabab29600 TRACE imxvpuapi imxvpuapi2_imx8m_hantro_encoder.c:1039:imx_vpu_api_enc_set_bitrate: setting bitrate to 2761338568 kbps

It is imx8mm + Linux 4.14.98 + gstreamer-imx v2.

Code is pretty straightforward - works on desktop with x264enc.

I believe I can't trace it deeper so I'd ask for help with that.

[dv1]

Normally, I'd consider the data type of the quantity (2000 in this case). g_object_set is not typesafe at compile time, since it uses C varargs, so I developed a habit of explicitly casting to match the type with the one from the GObject property, like this:

g_object_set(G_OBJECT(videnc), "bitrate", guint(2000), nullptr);

However - "2000" is an int, so it should not matter here.. I'll try it on my machine.

[dv1]

I found out what's wrong. It was a bug in libimxvpuapi. See https://github.com/Freescale/libimxvpuapi/commit/4c682f1330ce166328859438e632580ca5125030

Can you test the latest libimxvpuapi git master?

[andrewvoznytsa]

@dv1 thank you for looking into that. I'll try it next week (in fact will try to find some room next days but quite a few urgent things here).

Looking on that patch - are you sure that OMX_Video_ControlRateVariable will produce CBR stream (constant bitrate stream) until next bitrate change request? (It is what I assumed to use)

On first guess OMX_Video_ControlRateVariable should activate VBR (variable bitrate) rate control which usually tries to keep constant video quality, allowing stream bitrate fluctuations. Variable bitrate (when encoder decides to change video bitrate itself, not due to our request) will be problematic to us - we use some RF link (radio modems) to stream video and we can't afford the case when bitrate goes over link budget (and VBR is this case - we can't guarantee bitrate).

Anyway - I'll test it on my side and get back to you. Thank you again!

[dv1]

I am not sure if the encoder actually can do proper CBR. Even if I set it to "Constant", I do not notice significant differences in output buffer sizes. I am not sure though. Will check.

[dv1]

Any results on your end? I unfortunately did not yet have the time to test the bitstream. In my last test, I again did not see meaningful differences between OMX_Video_ControlRateVariable and OMX_Video_ControlRateConstant modes. I suspect that the Hantro driver treats them mostly the same..?

[andrewvoznytsa]

@dv1 sorry for delays - a lot of things happened here.

I did the tests (very basic) today and as for me it looks quite adequate. Thank you for your support! I think it would take me ages to find such solution.

[dv1]

Great :) Just one request if possible - can you run a basic test with OMX_Video_ControlRateConstant instead of OMX_Video_ControlRateVariable and let me know what differences you see, if any? Intuitively, it seems to me that OMX_Video_ControlRateConstant would be the "correct" choice, but as said, I see little actual difference.

[andrewvoznytsa]

On first look OMX_Video_ControlRateVariable is more smooth on quantizer and probably does not drop frames (bitrate change is relatively smooth). OMX_Video_ControlRateConstant is more aggressive on keeping bitrate within some window. Perhaps OMX_Video_ControlRateVariable does averaging over longer window and is kind of ABR. But honestly - it is just guess - I looked on bitrate graph in Wireshark and that's all. I expect to get more time for my bitrate exercises later and most likely will have time to evaluate RD plots and compare them.

BTW, do you plan to enhance encoder API a bit? It would be handy to be able to force key frame when we need that and to control RC buffer size together with IDR QP ratio (if we set relatively large GOP length encoder 'feels free' to allocate too much for IDR frame and we have bitrate burst - radio is not happy then).

[dv1]

I'd be very happy to hear suggestions for how to enhance the encoding API, since I don't do much encoding and have no real-world hands-on experience in that regard. But - I want to make a release soon (latest on this weekend), so if you have suggestions, please post them now, otherwise they'll have to make it into the next release. This concerns both libimxvpuapi and gstreamer-imx.

Also, note that not everything is supported by all encoders. I'll check what you suggested (key frames, RC buffer size etc.) In the h264 case, would forcing an IDR frame be sufficient? Nevermind, will experiment with that.

[dv1]

Small update: I did not find yet controls for the IDR QP ratio. Neither did I see a generic RC buffer control. But I did find VBV buffer size controls. Would these be useful for you?

[andrewvoznytsa]

Yes, VBV buffer size control is what people usually needs. Attached are my simple test apps to test dynamic bitrate and key frame on demand. Hope this key frame on demand will be useful for you. I see that key frame forcing is actually implemented on gstreamer-imx side (https://github.com/Freescale/gstreamer-imx/blob/master/src/vpu/encoder_base.c#L700) but for some reason it does not work.

Dynamic bitrate change + VBV control + key frame on demand is gentleman's set of futures to make that encoder working good in live environment. I believe anyone who is doing something more complex that 'hello world live streaming' will appetite that.

gst-imx8mm-examples.tar.gz

[andrewvoznytsa]

Actually I did not test key frame forcing after switching to OMX_Video_ControlRateVariable. In this strange world I won't be surprised that it will affect key frames. Will do that later today.

[dv1]

Unfortunately, the VBV buffer size control in the Hantro driver goes nowhere. There is a CPB buffer size control, but it is commented out, with the extra comment: // NOTE! Do not change unless you really know what you are doing!

Enabling this seems to help, but I can't be sure that it doesn't cause adverse effects. Also, it requires patching the imx-vpu-hantro package.

However, I was able to add code for forcing I/IDR frames via this flag: https://gstreamer.freedesktop.org/documentation/video/gstvideoutils.html?gi-language=c#GST_VIDEO_CODEC_FRAME_SET_FORCE_KEYFRAME

[andrewvoznytsa]

Thank you for the key frames! Can you please point me to this CBP buffer size control, if this is open source? I wrote a few rate controls in the past so perhaps I'll find a way to use it (and will submit PR for your review at the end).

[dv1]

It is contained in the imx-vpu-hantro package that is downloaded by the meta-freescale recipe of the same name. Inside, there's the file openmax_il/source/encoder/codec.h . In there, there's a RATE_CONTROL_CONFIG structure, which contains the commented-out hrdCpbSize field.

Note however that I had to turn off HRD to enable dynamic bitrate adjustments, so I am not sure if this field would help.

The imx-vpu-hantro code is a mix of an OpenMAX IL interface with vendor extras & some internal code that is below that interface, so this can be quite confusing.

[andrewvoznytsa]

There is in fact different way to make dynamic bitrate adjustments working - I can create GstBin, put there two IMX encoders (one 'current' and second 'next to be used') and switch between them in run time. This 'next to be used' can be stopped/configured/started where needed. That GstBin can expose all required encoder controls and act just like an encoder element with some tricky internals. And in this case enabling HRD would be actually good option.

But I like your current approach and will stick with it. If I manage to get any rate control improvement then I'll submit them here.

[dv1]

That could work, but would require switching at GOP granularities, since otherwise, references to previous frames would cause problems I guess.

As for additions, I think I'll have to postpone those to the next libimxvpuapi and gstreamer-imx release, since I need to make one very soon. That way, there's more time later for a thorough analysis.