Caught a segmentation fault while loading plugin file: gstreamer-1.0/libgstlibav.so

[bbjay]

Hello,

I had a working installation on Ubuntu 18.04, but after an upgrade to 19.10 I'm getting:

ERROR: Caught a segmentation fault while loading plugin file:
/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstlibav.so

Please either:
- remove it and restart.
- run with --gst-disable-segtrap --gst-disable-registry-fork and debug.
Segmentation fault (core dumped)

I've tried it with with obs-studio packages from Ubuntu and with the one from ppa:obsproject/obs-studio. Also I've tried to build obs-gstreamer from source (v0.0.6 and v0.1.0) with the same result. Even after reinstalling 18.04 I can't get it to work anymore.

The pipeline I'm using is udpsrc port=5001 ! h264parse ! avdec_h264 ! video. and launching it with gst-launch-1.0 udpsrc port=5001 ! h264parse ! avdec_h264 ! autovideosink sync=false works.

Any ideas on what to do here?

[fzwoch]

It sounds to me you have conflicting versions of FFMPEG (and/or GStreamer) installed somewhere. Of course running it under gdb may give some hints where it is crashing. The plugin itself does not operate with FFMPEG directly - just with a very high level GStreamer interface that should be stable along the 1.x path.

Try gst-inspect-1.0 libav and check the output carefully. Does it blacklist plugins? Is there any error? It may be that one particular element from libav is not working - the GStreamer tools may still continue as long as you don't use this specific plugin while OBS may be more strict and just stops execution.

Then again maybe OBS does come with its own copy of FFMPEG incompatible with the system's which GStreamer is linked to. I usually build my OBS myself so I'm pretty certain both parts use the same for me..

[bbjay]

Thanks for your reply. I checked your suggestions and even learned some new debugging techniques :-)

The gst-inspect-1.0 libav didn't show anything suspicious.

So I went building obs-studio myself (was easier than I thought, the instructions on their website are really good), uninstalled the current one to be sure which one was used, and rebuilt obs-gstreamer from source with the new libobs.

Unfortunately the segfault still happened. There were only the Ubuntu system ffmpeg and gstreamer packages installed.

So I started debugging and after installing the debug symbol packages I got the following output:

$ gdb --args obs --gst-disable-segtrap --gst-disable-registry-fork
...
Thread 11 "libobs: graphic" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffabfff700 (LWP 14714)]
0x00007fff574f95b3 in gst_ffmpegdemux_register (plugin=plugin@entry=0x555556481640) at gstavdemux.c:1990
1990    gstavdemux.c: No such file or directory.
(gdb) backtrace 
#0  0x00007fff574f95b3 in gst_ffmpegdemux_register (plugin=plugin@entry=0x555556481640) at gstavdemux.c:1990
#1  0x00007fff574db2c2 in plugin_init (plugin=0x555556481640) at gstav.c:165
#2  0x00007fffa947a445 in gst_plugin_register_func (plugin=0x555556481640, desc=0x7fff57709f80 <gst_plugin_desc>, user_data=0x0) at gstplugin.c:524
#3  0x00007fffa947c39e in _priv_gst_plugin_load_file_for_registry (filename=0x5555564b8c40 "/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstlibav.so", registry=0x555556408130, registry@entry=0x0, error=error@entry=0x7fffabffcf00) at gstplugin.c:885
#4  0x00007fffa947cdaa in gst_plugin_load_file (filename=<optimized out>, error=error@entry=0x7fffabffcf00) at gstplugin.c:681
#5  0x00007fffa947d1fc in gst_plugin_load_by_name (name=0x55555648c591 "libav") at gstplugin.c:1325
#6  0x00007fffa947db9e in gst_plugin_feature_load (feature=feature@entry=0x5555563d3680) at gstpluginfeature.c:112
#7  0x00007fffa945436e in gst_element_factory_create (factory=factory@entry=0x5555563d3680, name=name@entry=0x0) at gstelementfactory.c:348
#8  0x00007fffa945469e in gst_element_factory_make (factoryname=0x7fffa002f1b0 "avdec_h264", name=name@entry=0x0) at gstelementfactory.c:445
#9  0x00007fffa94c7250 in priv_gst_parse_yyparse (scanner=<optimized out>, graph=graph@entry=0x7fffabffe600) at ./grammar.y:813
#10 0x00007fffa94c8118 in priv_gst_parse_launch (str=str@entry=0x7fffa000f280 "videoconvert name=video ! video/x-raw, format={I420,NV12,BGRA,BGRx,RGBx,RGBA,YUY2,YVYU,UYVY} ! appsink name=video_appsink audioconvert name=audio ! audioresample ! audio/x-raw, format={U8,S16LE,S32LE,"..., error=error@entry=0x7fffabffe680, ctx=ctx@entry=0x0, flags=flags@entry=GST_PARSE_FLAG_NONE) at ./grammar.y:1177
#11 0x00007fffa94bee5f in gst_parse_launch_full (pipeline_description=0x7fffa000f280 "videoconvert name=video ! video/x-raw, format={I420,NV12,BGRA,BGRx,RGBx,RGBA,YUY2,YVYU,UYVY} ! appsink name=video_appsink audioconvert name=audio ! audioresample ! audio/x-raw, format={U8,S16LE,S32LE,"..., context=0x0, flags=GST_PARSE_FLAG_NONE, error=0x7fffabffe6d0) at gstparse.c:339
#12 0x00007fffa9731ef9 in start (data=0x555556938190) at ../gstreamer.c:263
#13 0x00007fffa9732646 in show (data=0x555556938190) at ../gstreamer.c:411
#14 0x00007ffff4c38d66 in show_source (source=0x555556904920) at /home/user/obs-studio/libobs/obs-source.c:954
#15 0x00007ffff4c3ec85 in obs_source_video_tick (source=source@entry=0x555556904920, seconds=seconds@entry=0.100000001)
    at /home/user/obs-studio/libobs/obs-source.c:1086
#16 0x00007ffff4c7c9ee in tick_sources (last_time=<optimized out>, cur_time=98500974186385) at /home/user/obs-studio/libobs/obs-video.c:65
#17 0x00007ffff4c7c9ee in obs_graphics_thread (param=<optimized out>) at /home/user/obs-studio/libobs/obs-video.c:874
#18 0x00007ffff31c66db in start_thread (arg=0x7fffabfff700) at pthread_create.c:463
#19 0x00007ffff2eef88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Frame 8 tells me that the crash happens when the avdec_h264 element is being created.

After a $ sudo apt-get remove gstreamer1.0-libav obs-studio launches successfully and I can also edit the obs-gstreamer source properties (but of course the video does not show then and the log says something about Gstreamer couldn't start because of missing element avdec_h264).

So my next step will be building gstreamer1.0-libav from source to see where and why exactly the segfault happens.

Or did I miss something obvious you could tell from the above?

[fzwoch]

Interesting. The real culprit really is on top:

#0  0x00007fff574f95b3 in gst_ffmpegdemux_register (plugin=plugin@entry=0x555556481640) at gstavdemux.c:1990

Would be interesting to check that code - if you can find the version you are running. It is failing due to avdec_h264, true - however what is actually crashing is the something in the demuxer element.

This seems to be collateral damage. You only want the decoder, but all FFMPEG functions are in one big plugin. So the demuxer pulls you into the abyss because it is crashing for some reason although you don't even want it.

Perhaps some optional package is missing on the system that slipped our attention?

I cant tell the reason tbh. If you have an stock Ubuntu GStreamer and obs package (or building yourself) both should be linked to the same FFMPEG libraries - so no conflict there. Perhaps you can do some more digging and we can figure it out.. :-)

[fzwoch]

Check this one:

https://gitlab.freedesktop.org/gstreamer/gst-libav/commit/9e56619b10e741dc7ffaba63c3001536fe3a8b66

This has been changed 3 weeks ago. It is not quite at that line number but close. At line 1990 there is actually no code. But maybe that's a hint. I have no idea why the regular gst-launch-1.0 call would not crash though. Luck perhaps? At least that fixes a segfault..

[bbjay]

Check this one:

Yes that was it!

Didn't even need to debug, just checked out this commit, built & installed it and it worked. One commit earlier and the segfault is there again. (ok it took me a while ;-) since i needed to upgrade to 19.10 again to satisfy the dependency libs versions of the new gst-libav)

Really strange that it ever worked on 18.04 in the first place, also that it still works with gst-launch... anyway

Many thanks for helping with this and for creating this awesome plugin!

[fzwoch]

Sweet, so we kinda "found" a bug in GStreamer, well, someone else found and fixed it - but it wasn't us. :-)

Perhaps the FFMPEG version was too new for the GStreamer library causing some issues. Why it did work before but not anymore remains a mystery. Maybe you installed additional packages you haven't before? Who knows - it works now - do something fun with it!