Framerate is Insane Warnings Despite Valid Frame Rate

[jesusvigil]

When using RGA for hardware accelerated video conversion, the mpph264enc element logs a WARN message indicating that the framerate 0/1 is insane!. The frame rate is specified in a cap at the element v4l2src which is being used to capture from the /dev/video0 device so it is understood that the frame rate is valid.

This framerate 0/1 is insane WARN message is logged for any frame rate that is fed to the mpph264enc element. GStreamer graphs also validate that the expected frame rate is being sent to mpph264enc.

It is believed that the issue may lie with the usage of this line gst_video_info_set_format (info, gst_mpp_mpp_format_to_gst_format (format), width, height); at line 650 of the gstmppenc.c file (link).

The function gst_video_info_set_format () appears to not preserve any values passed in from the info object according to the GStreamer documentation linked here.

Because converting with RGA is enabled, we enter a logic path where the function gst_video_info_set_format () is called and that clears all values in the info object. The framerate data gets cleared thus causing the WARN log framerate 0/1 is insane! and the framerate to always default to the value of DEFAULT_FPS.

Could it be possible to temporarily save the frame rate data before this gst_video_info_set_format () function is called and add the data back to the info object after the function returns?

[JeffyCN]

right, please try: https://github.com/JeffyCN/mirrors/commit/53fc0a822001b7dbdb6c0d5286b9904c4f1db3be

[jesusvigil]

Thank you very much for looking into this issue. I've tested this and it appears that GStreamer is catching a SIGSEGV. I've added additional logging to try to narrow down where the error occurs and I believe it is at line 340 of your new changes in the file gstmpp.c: gst_caps_set_simple (caps, "format", G_TYPE_STRING, format, "width", G_TYPE_INT, width, "height", G_TYPE_INT, height, NULL);

Please let me know if you need more information.

[JeffyCN]

re-uploaded, should be ok now

[jesusvigil]

Thank you very much for the fix! I've tested this and all looks well.

[jesusvigil]

After additional testing, it appears that MP4 files are not able to play back using QuickTime player or using gst-play-1.0 (GStreamer 1.18.5). We have the following gst-play-1.0 logs:

WARNING Could not initialise Wayland output
WARNING debug information: ../ext/wayland/gstwaylandsink.c(503): gst_wayland_sink_find_display (): /GstWaylandSink:waylandsink0:
Failed to create GstWlDisplay: 'Failed to connect to the wayland display '(default)''

** (gst-play-1.0:8069): CRITICAL **: 16:51:28.744: gst_video_info_to_caps: assertion 'info->finfo != NULL' failed

(gst-play-1.0:8069): GStreamer-CRITICAL **: 16:51:28.744: gst_caps_set_simple: assertion 'GST_IS_CAPS (caps)' failed

** (gst-play-1.0:8069): CRITICAL **: 16:51:28.744: gst_video_info_from_caps: assertion 'caps != NULL' failed

(gst-play-1.0:8069): GStreamer-CRITICAL **: 16:51:28.744: gst_mini_object_unref: assertion 'mini_object != NULL' failed
Segmentation fault

I see that there is a commit on May 29, 2023 and have built that and tested it with mpph264enc which yielded slightly different errors with gst-play-1.0, but I believe may be related.

WARNING Could not initialise Wayland output
WARNING debug information: ../ext/wayland/gstwaylandsink.c(503): gst_wayland_sink_find_display (): /GstWaylandSink:waylandsink0:
Failed to create GstWlDisplay: 'Failed to connect to the wayland display '(default)''

** (gst-play-1.0:8137): CRITICAL **: 16:54:18.788: gst_video_info_to_caps: assertion 'info->finfo->format != GST_VIDEO_FORMAT_UNKNOWN' failed

(gst-play-1.0:8137): GStreamer-CRITICAL **: 16:54:18.789: gst_caps_set_simple: assertion 'GST_IS_CAPS (caps)' failed

** (gst-play-1.0:8137): CRITICAL **: 16:54:18.789: gst_video_info_from_caps: assertion 'caps != NULL' failed

(gst-play-1.0:8137): GStreamer-CRITICAL **: 16:54:18.789: gst_mini_object_unref: assertion 'mini_object != NULL' failed

** (gst-play-1.0:8137): CRITICAL **: 16:54:18.789: gst_mpp_dec_update_video_info: assertion 'format != GST_VIDEO_FORMAT_UNKNOWN' failed

Please let me know if further information is needed.

[JeffyCN]

try: https://github.com/JeffyCN/mirrors/commit/1a9a34d142d0ff708788f880acaf5a74adc3b014

[jesusvigil]

Thank you for the reply. It appears that the issue still persists. When I attempt to playback the video using the Videos application on Debian, I receive an internal data stream error. Below is logging from gst-play-1.0:

0:00:00.163924048  4209   0x7f941db4c0 INFO               baseparse gstbaseparse.c:4859:gst_base_parse_set_upstream_tags:<h264parse0> upstream tags: taglist, video-codec=(string)H264;
0:00:00.163987633  4209   0x7f941db6a0 INFO            audiodecoder gstaudiodecoder.c:2522:gst_audio_decoder_sink_eventfunc:<avdec_aac0> upstream stream tags: taglist, audio-codec=(string)"MPEG-4\ AAC", language-code=(string)en, title=(string)Audio;
0:00:00.164014175  4209   0x7f941db4c0 INFO               baseparse gstbaseparse.c:4083:gst_base_parse_set_latency:<h264parse0> min/max latency 0:00:00.000000000, 0:00:00.000000000
0:00:00.164151261  4209   0x7f941db4c0 INFO            videodecoder gstvideodecoder.c:1448:gst_video_decoder_sink_event_default:<mppvideodec0> upstream tags: taglist, video-codec=(string)H264;
0:00:00.164188595  4209   0x7f941db4c0 INFO            videodecoder gstvideodecoder.c:1448:gst_video_decoder_sink_event_default:<mppvideodec0> upstream tags: taglist, video-codec=(string)H264;
0:00:00.164252763  4209   0x7f941db4c0 INFO               videometa gstvideometa.c:1072:gst_video_time_code_meta_api_get_type: registering
0:00:00.164279014  4209   0x7f941db4c0 INFO            videodecoder gstvideodecoder.c:1448:gst_video_decoder_sink_event_default:<mppvideodec0> upstream tags: taglist, video-codec=(string)"H.264\ \(High\ Profile\)";
0:00:00.164309639  4209   0x7f941db6a0 INFO               GST_EVENT gstevent.c:847:gst_event_new_caps: creating caps event audio/x-raw, format=(string)F32LE, layout=(string)non-interleaved, rate=(int)48000, channels=(int)2, channel-mask=(bitmask)0x0000000000000003
0:00:00.164349307  4209   0x7f941db4c0 INFO        GST_ELEMENT_PADS gstelement.c:1009:gst_element_get_static_pad: found pad xvimagesink0:sink
0:00:00.164510018  4209   0x7f941db6a0 INFO                GST_PADS gstpad.c:4336:gst_pad_peer_query:<'':decodepad2> pad has no peer
0:00:00.164715064  4209   0x7f941db4c0 INFO           basetransform gstbasetransform.c:1325:gst_base_transform_setcaps:<capsfilter0> reuse caps
0:00:00.164795565  4209   0x7f941db4c0 INFO                    task gsttask.c:516:gst_task_set_lock: setting stream lock 0x7f8c026a00 on task 0x7f84005050
0:00:00.164813066  4209   0x7f941db4c0 INFO                GST_PADS gstpad.c:6270:gst_pad_start_task:<mppvideodec0:src> created task 0x7f84005050
0:00:00.167016651  4209   0x7f8c017700 INFO                  mppdec gstmppdec.c:493:gst_mpp_dec_apply_info_change:<mppvideodec0> applying NV12 1920x1080 (1920x1088)

** (gst-play-1.0:4209): CRITICAL **: 20:15:18.901: gst_video_info_to_caps: assertion 'info->finfo->format != GST_VIDEO_FORMAT_UNKNOWN' failed

(gst-play-1.0:4209): GStreamer-CRITICAL **: 20:15:18.901: gst_caps_set_simple: assertion 'GST_IS_CAPS (caps)' failed

** (gst-play-1.0:4209): CRITICAL **: 20:15:18.901: gst_video_info_from_caps: assertion 'caps != NULL' failed

(gst-play-1.0:4209): GStreamer-CRITICAL **: 20:15:18.901: gst_mini_object_unref: assertion 'mini_object != NULL' failed
0:00:00.167181737  4209   0x7f8c017700 INFO                  mppdec gstmppdec.c:512:gst_mpp_dec_apply_info_change:<mppvideodec0> convert from NV12 (1920x1080) to UNKNOWN (0x0)

** (gst-play-1.0:4209): CRITICAL **: 20:15:18.901: gst_mpp_dec_update_video_info: assertion 'format != GST_VIDEO_FORMAT_UNKNOWN' failed
0:00:00.167219655  4209   0x7f8c017700 INFO                  mppdec gstmppdec.c:917:gst_mpp_dec_loop:<mppvideodec0> video info changed
0:00:00.167235988  4209   0x7f8c017700 INFO                    task gsttask.c:368:gst_task_func:<mppvideodec0:src> Task going to paused
0:00:00.173470486  4209   0x7f941db4c0 INFO                    task gsttask.c:368:gst_task_func:<multiqueue0:src_0> Task going to paused

[JeffyCN]

please try this: https://github.com/JeffyCN/mirrors/commit/d5cf5d7d384955849e41c93f61eeaad11eca6157

[jesusvigil]

Thank you very much for your efforts towards this. I have tested this and files are able to encode and playback with no problems.