`add_metadata_event_struct` causes segmentation fault immediately

[mkaivs]

I built and installed gst-shark on Ubuntu 20.04 as follow:

sudo apt install graphviz libgraphviz-dev
sudo apt install octave epstool babeltrace
sudo apt install gtk-doc-tools
git clone https://github.com/RidgeRun/gst-shark/
cd gst-shark
./autogen.sh --prefix /usr/ --libdir /usr/lib/x86_64-linux-gnu/
make
sudo make install

This issue doesn't occur when I build from this tag: https://github.com/RidgeRun/gst-shark/tree/v0.6.0. The issue occurred when I use the current master branch.

Then environment variables are: GST_SHARK_FILE_BUFFERING=0, GST_DEBUG="GST_TRACER:7", GST_TRACERS="interlatency"

The output:

0:00:00.003817115 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:171:gst_tracer_register:<tracerfactory0> new tracer factory for cpuusage
0:00:00.003859069 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:178:gst_tracer_register:<cpuusage> tracer factory for 1939735280:GstCPUUsageTracer
0:00:00.003879057 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:171:gst_tracer_register:<tracerfactory1> new tracer factory for graphic
0:00:00.003890783 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:178:gst_tracer_register:<graphic> tracer factory for 1939741872:GstGraphicTracer
0:00:00.003915378 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:171:gst_tracer_register:<tracerfactory2> new tracer factory for proctime
0:00:00.003931716 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:178:gst_tracer_register:<proctime> tracer factory for 1939743232:GstProcTimeTracer
0:00:00.003954616 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:171:gst_tracer_register:<tracerfactory3> new tracer factory for interlatency
0:00:00.003965952 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:178:gst_tracer_register:<interlatency> tracer factory for 1939744192:GstInterLatencyTracer
0:00:00.003986165 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:171:gst_tracer_register:<tracerfactory4> new tracer factory for scheduletime
0:00:00.003997444 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:178:gst_tracer_register:<scheduletime> tracer factory for 1939745216:GstScheduletimeTracer
0:00:00.004017877 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:171:gst_tracer_register:<tracerfactory5> new tracer factory for framerate
0:00:00.004029255 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:178:gst_tracer_register:<framerate> tracer factory for 1939745904:GstFramerateTracer
0:00:00.004057913 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:171:gst_tracer_register:<tracerfactory6> new tracer factory for queuelevel
0:00:00.004070310 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:178:gst_tracer_register:<queuelevel> tracer factory for 1939747008:GstQueueLevelTracer
0:00:00.004184766 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:171:gst_tracer_register:<tracerfactory7> new tracer factory for bitrate
0:00:00.004203239 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:178:gst_tracer_register:<bitrate> tracer factory for 1939747424:GstBitrateTracer
0:00:00.004234651 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:171:gst_tracer_register:<tracerfactory8> new tracer factory for buffer
0:00:00.004247250 35366 0x55e7739d6a90 DEBUG             GST_TRACER gsttracer.c:178:gst_tracer_register:<buffer> tracer factory for 1939751584:GstBufferTracer
0:00:00.103018326 35362 0x56471f920800 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0x56471fb70be0 (cpuusage)
0:00:00.103047619 35362 0x56471f920800 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0x56471fb70b20 (graphic)
0:00:00.103057431 35362 0x56471f920800 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0x56471fb70a60 (proctime)
0:00:00.103065573 35362 0x56471f920800 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0x56471fb709a0 (interlatency)
0:00:00.103073369 35362 0x56471f920800 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0x56471fb708e0 (scheduletime)
0:00:00.103083432 35362 0x56471f920800 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0x56471fb70820 (framerate)
0:00:00.103091245 35362 0x56471f920800 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0x56471f92aef0 (queuelevel)
0:00:00.103100717 35362 0x56471f920800 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0x56471f92ae30 (bitrate)
0:00:00.103108526 35362 0x56471f920800 DEBUG             GST_TRACER gsttracer.c:161:gst_tracer_register:<registry0> update existing feature 0x56471f92ad70 (buffer)
0:00:00.103163260 35362 0x56471f920800 TRACE             GST_TRACER gsttracerrecord.c:111:gst_tracer_record_build_format: interlatency.class, from_pad=(structure)"scope\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PAD\;", to_pad=(structure)"scope\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PAD\;", time=(structure)"scope\,\ type\=\(type\)gchararray\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;";
0:00:00.103177431 35362 0x56471f920800 DEBUG             GST_TRACER gsttracerrecord.c:125:gst_tracer_record_build_format: new format string: interlatency, from_pad=(string)%s, to_pad=(string)%s, time=(string)%s;
Segmentation fault (core dumped)

The trace:

add_metadata_event_struct is the function that caused the Segmentation fault.

The issue can be reproduced by running this command:

GST_DEBUG="GST_TRACER:7" GST_TRACERS="interlatency" gst-launch-1.0 videotestsrc ! queue ! videorate max-rate=15 ! fakesink sync=true

[alehed]

I can reproduce this issue. Currently interlatency, proctime, queuelevel and scheduletime are unusable due to this. framerate, cputime and bitrate seem to work.

[MichaelOultram-pexip]

Running one of the example pipelines under gdb: GST_DEBUG="GST_TRACER:7" GST_TRACERS="queuelevel" gdb gst-launch-1.0 videotestsrc ! 'video/x-raw, format=(string)YUY2, width=(int)640, height=(int)480, framerate=(fraction)30/1' ! videorate max-rate=30 ! videoconvert ! queue max-size-buffers=20 ! avenc_h263p ! queue max-size-time=400000000 ! avimux ! fakesink sync=true

(gdb) bt
#0  0x00007ffff7db0159 in g_mutex_lock () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#1  0x00007ffff6e4a3df in add_metadata_event_struct
    (metadata_event=0x555555693000 "event {\n    name = queuelevel;\n    id = 6;\n    stream_id = 0;\n    fields := struct {\n        string queue;\n        integer { size = 32; align = 8; signed = 0; encoding = none; base = 10; } size_bytes;"...)
    at ../plugins/tracers/gstctf.c:744
#2  0x00007ffff7fa5db1 in gst_queue_level_tracer_class_init (klass=0x55555587f710) at ../plugins/tracers/gstqueuelevel.c:207
#3  0x00007ffff7fa5390 in gst_queue_level_tracer_class_intern_init (klass=0x55555587f710) at ../plugins/tracers/gstqueuelevel.c:41
#4  0x00007ffff7ce9b28 in g_type_class_ref () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#5  0x00007ffff7cd2baa in g_object_new_valist () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#6  0x00007ffff7cd2c8d in g_object_new () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#7  0x00007ffff7e8516f in  () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#8  0x00007ffff7d6fa38 in g_option_context_parse () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#9  0x000055555555851f in  ()
#10 0x00007ffff7a29d90 in __libc_start_call_main (main=main@entry=0x5555555581c0, argc=argc@entry=1, argv=argv@entry=0x7fffffffd978) at ../sysdeps/nptl/libc_start_call_main.h:58
#11 0x00007ffff7a29e40 in __libc_start_main_impl (main=0x5555555581c0, argc=1, argv=0x7fffffffd978, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7fffffffd968) at ../csu/libc-start.c:392
#12 0x0000555555558c05 in _start ()
(gdb) f 1
#1  0x00007ffff6e4a3df in add_metadata_event_struct (
    metadata_event=0x555555693000 "event {\n    name = queuelevel;\n    id = 6;\n    stream_id = 0;\n    fields := struct {\n        string queue;\n        integer { size = 32; align = 8; signed = 0; encoding = none; base = 10; } size_bytes;"...)
    at ../plugins/tracers/gstctf.c:744
744   g_mutex_lock (&ctf_descriptor->mutex);
(gdb) p ctf_descriptor
$1 = (GstCtfDescriptor *) 0x0
(gdb) 

It seems like the gst_ctf_init isn't being called. Looking at the diff between v0.6.0 and v0.8.0, I found a change to plugins/tracers/gstplugin.c in 11e6f12f5f17e68c8e6fd938660e74edcc8aa767

diff --git a/plugins/tracers/gstplugin.c b/plugins/tracers/gstplugin.c
index 900c4df..63efe1c 100644
--- a/plugins/tracers/gstplugin.c
+++ b/plugins/tracers/gstplugin.c
@@ -74,9 +74,6 @@ plugin_init (GstPlugin * plugin)
   if (!gst_tracer_register (plugin, "buffer", gst_buffer_tracer_get_type ())) {
     return FALSE;
   }
-  if (!gst_ctf_init ()) {
-    return FALSE;
-  }

   return TRUE;
 }

Reverting that change stops the segmentation fault for me. I don't know this codebase enough to know if this is the correct fix, but it allows me to at least run what I wanted to.

[vebjornjr]

I get the same behaviour on Ubuntu 18.04. The fix from @MichaelOultram-pexip seems to work, thank you!

[jh-aug]

I had the same problem on Xavier NX running Ubuntu 20.04 with Nvidia patches on Gstreamer 1.16.3. Great find by @MichaelOultram-pexip fixed the issue for me. Many thanks!

[michaelgruner]

Hey all, thanks for the reports and fixes! Sorry for the slow response. I've published a fix for this in the latest release. Please feel free to re-open if the problem persists.