Segfault on video streaming to Intel GPU on Linux

AndKe commented 3 years ago

Expected Behavior

"This should just work."

Current Behavior

The QGC crashes instantly, before displaying any video, when attempting to do so.

Steps to Reproduce:

On Ubuntu with intel video , start a video stream.
observe the segfault.

System Information

-Ubuntu 20.10

QGC Version: verified with current HEAD (fresh local built) , and latest .appimage
Flight Controller: not relevant, but Disco and solo crashes equally well

-Desktop PC GPU : 00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630 (Desktop 9 Series) (rev 02) 00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model

-Laptop PC GPU: 00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT Integrated Graphics Controller (rev 09)

Detailed Description

This bug has existed for at least half a year. Both computers can stream the video just fine with "any" other app that can recieve the stream, tested with smplayer, vlc, gstreamer

Log Files and Screenshots

$ ./QGroundControl.AppImage 
Settings location "/home/andre/.config/QGroundControl.org/QGroundControl Daily.ini" Is writable?: true
Filter rules "*Log.debug=false\nGStreamerLog.debug=true\nLinkManagerLog.debug=true\n"
GStreamerLog: Disable bcmdec
System reported locale: QLocale(English, Latin, United States) ; Name "en_US" ; Preffered (used in maps):  "en-US"
MAVLinkLogManagerLog: MAVLink logs directory: "/home/andre/Documents/QGroundControl Daily/Logs"
Map Cache in: "/home/andre/.cache/QGCMapCache300" / "qgcMapCache.db"
qml: QGCCorePlugin(0x56046df8e490) []
qrc:/qml/MainRootWindow.qml:412:13: QML ColumnLayout: Possible anchor loop detected on vertical anchor.
LinkManagerLog: New auto-connect UDP port added
Adding target QHostAddress("10.1.1.1") 47909
Temp log "/tmp/FlightData130616.mavlink"
libpng warning: iCCP: known incorrect sRGB profile
Segmentation fault (core dumped)

DonLakeFlyer commented 3 years ago

@AndKe Also if you have built yourself a stack trace would be great.

AndKe commented 3 years ago

@DonLakeFlyer Of course, please see below :) FWIW : one does not need a working stream with video (no camera need to be connected, just a black/empty stream is sufficient.)

andre@Aranand:~/prog/qgroundcontrol/build/staging$ gdb ./QGroundControl 
GNU gdb (Ubuntu 9.2-0ubuntu2) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./QGroundControl...
r
(gdb) r
Starting program: /home/andre/prog/qgroundcontrol/build/staging/QGroundControl 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffee29c640 (LWP 170759)]
[New Thread 0x7fffe7fff640 (LWP 170760)]
[New Thread 0x7fffe77fe640 (LWP 170761)]
[New Thread 0x7fffe6ffd640 (LWP 170762)]
Settings location "/home/andre/.config/QGroundControl.org/QGroundControl Daily.ini" Is writable?: true
Filter rules "*Log.debug=false\nGStreamerLog.debug=true\nLinkManagerLog.debug=true\n"
GStreamerLog: Disable bcmdec
System reported locale: QLocale(English, Latin, United States) ; Name "en_US" ; Preffered (used in maps):  "en-US"
[Detaching after fork from child process 170764]
[New Thread 0x7fffe6647640 (LWP 170791)]
serialnmea: No known GPS device found. Specify the COM port via QT_NMEA_SERIAL_PORT.
[New Thread 0x7fffe4efc640 (LWP 170806)]
[New Thread 0x7fffd7fff640 (LWP 170807)]
MAVLinkLogManagerLog: MAVLink logs directory: "/home/andre/Documents/QGroundControl Daily/Logs"
Map Cache in: "/home/andre/.cache/QGCMapCache300" / "qgcMapCache.db"
[New Thread 0x7fffd77fe640 (LWP 170808)]
[New Thread 0x7fffd6ffd640 (LWP 170809)]
[New Thread 0x7fffd5ee8640 (LWP 170810)]
[New Thread 0x7fffb4166640 (LWP 170811)]
[New Thread 0x7fffb3965640 (LWP 170812)]
[New Thread 0x7fffb3164640 (LWP 170813)]
[New Thread 0x7fffb2963640 (LWP 170814)]
[New Thread 0x7fffb2142640 (LWP 170815)]
qml: QGCCorePlugin(0x555557878370) []
[New Thread 0x7fff9bfff640 (LWP 170816)]
[New Thread 0x7fff9b3ac640 (LWP 170817)]
qrc:/qml/MainRootWindow.qml:412:13: QML ColumnLayout: Possible anchor loop detected on vertical anchor.
[New Thread 0x7fff9abab640 (LWP 170818)]
LinkManagerLog: New auto-connect UDP port added
[New Thread 0x7fff9a3aa640 (LWP 170819)]
[New Thread 0x7fff99ba9640 (LWP 170820)]
Adding target QHostAddress("10.1.1.1") 37817
[Thread 0x7fffd77fe640 (LWP 170808) exited]
Temp log "/tmp/FlightData540144.mavlink"
libpng warning: iCCP: known incorrect sRGB profile
[New Thread 0x7fffd77fe640 (LWP 171236)]
[New Thread 0x7fff83fff640 (LWP 171237)]
[New Thread 0x7fff7bfff640 (LWP 171238)]
[New Thread 0x7fff837fe640 (LWP 171239)]
[New Thread 0x7fff82ffd640 (LWP 171240)]
[New Thread 0x7fff81f56640 (LWP 171241)]
[New Thread 0x7fff80c0c640 (LWP 171243)]
[Thread 0x7fff80c0c640 (LWP 171243) exited]
[New Thread 0x7fff80c0c640 (LWP 171244)]
[New Thread 0x7fff7b7fe640 (LWP 171245)]
[New Thread 0x7fff7affd640 (LWP 171246)]
[New Thread 0x7fff7a7fc640 (LWP 171247)]
[New Thread 0x7fff79ffb640 (LWP 171248)]
[New Thread 0x7fff797fa640 (LWP 171249)]
[New Thread 0x7fff78ff9640 (LWP 171250)]

Thread 23 "queue0:src" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff83fff640 (LWP 171237)]
0x00007fff80e05bc5 in ?? () from /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
(gdb) bt
#0  0x00007fff80e05bc5 in  () at /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
#1  0x00007fff80df6bf4 in  () at /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
#2  0x00007fff80ddd61a in  () at /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
#3  0x00007fff80feb5e3 in  () at /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
#4  0x00007fff80fef016 in  () at /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
#5  0x00007fff80fef29a in  () at /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
#6  0x00007fff80fbcf92 in  () at /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
#7  0x00007fff9800f2b0 in vaPutImage () at /usr/lib/x86_64-linux-gnu/libva.so.2
#8  0x00007fff813dea14 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvaapi.so
#9  0x00007fff8139c9c6 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvaapi.so
#10 0x00007fff813a3cce in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvaapi.so
#11 0x00007ffff7c96c33 in  () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#12 0x00007ffff7c9a64d in  () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#13 0x00007ffff7b8f208 in gst_pad_query () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#14 0x00007ffff7b8f96b in gst_pad_peer_query () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#15 0x00007ffff7bcf3c8 in gst_pad_peer_query_caps () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#16 0x00007ffff7c9a32f in  () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#17 0x00007ffff7b8f208 in gst_pad_query () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#18 0x00007ffff7b8f96b in gst_pad_peer_query () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#19 0x00007ffff7bc95fc in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#20 0x00007ffff7b8db5e in gst_pad_forward () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#21 0x00007ffff7bcc29a in gst_pad_proxy_query_caps () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#22 0x00007ffff7b8dd88 in gst_pad_query_default () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#23 0x00007ffff7b8f208 in gst_pad_query () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#24 0x00007ffff7b8f96b in gst_pad_peer_query () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#25 0x00007ffff7bcf3c8 in gst_pad_peer_query_caps () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#26 0x00007ffff7d1f631 in  () at /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0
#27 0x00007fff813964f8 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvaapi.so
#28 0x00007ffff7d0da66 in  () at /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0
#29 0x00007ffff7b8f208 in gst_pad_query () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#30 0x00007ffff7bccb38 in gst_pad_query_caps () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#31 0x00007ffff7d0dcd1 in  () at /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0
#32 0x00007ffff7b8f208 in gst_pad_query () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#33 0x00007ffff7b8f96b in gst_pad_peer_query () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#34 0x00007ffff7bc9594 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#35 0x00007ffff7b8db5e in gst_pad_forward () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#36 0x00007ffff7bcc0ce in gst_pad_proxy_query_accept_caps () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#37 0x00007ffff7b8e35b in gst_pad_query_default () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#38 0x00007ffff7b8f208 in gst_pad_query () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#39 0x00007ffff7bcf5a8 in gst_pad_query_accept_caps () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#40 0x00007ffff7b87b4a in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#41 0x00007ffff7b91ba9 in gst_pad_send_event () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#42 0x00007fff9817d11c in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#43 0x00007ffff7b866dc in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#44 0x00007ffff7b85ef0 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
--Type <RET> for more, q to quit, c to continue without paging--c
#45 0x00007ffff7b927bc in gst_pad_sticky_events_foreach () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#46 0x00007fff98188a2f in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#47 0x00007fff98186625 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#48 0x00007fff9818717c in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#49 0x00007fff9818741e in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#50 0x00007ffff7ab68fa in g_closure_invoke () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#51 0x00007ffff7ac94b3 in  () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#52 0x00007ffff7acfc41 in g_signal_emit_valist () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#53 0x00007ffff7acfda3 in g_signal_emit () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#54 0x00007ffff7abb464 in  () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#55 0x00007ffff7b3de88 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#56 0x00007ffff7abd49a in g_object_notify_by_pspec () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#57 0x00007ffff7b865e7 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#58 0x00007ffff7b91820 in gst_pad_push_event () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#59 0x00007ffff7c99bf8 in  () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#60 0x00007ffff7c9b22a in  () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#61 0x00007fff990e9e96 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstcoreelements.so
#62 0x00007ffff7b87a7c in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#63 0x00007ffff7b880fe in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#64 0x00007ffff7b885b8 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#65 0x00007ffff7b85ef0 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#66 0x00007ffff7b91530 in gst_pad_push_event () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#67 0x00007fff98e09ef0 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvideoparsersbad.so
#68 0x00007fff98e0d20f in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvideoparsersbad.so
#69 0x00007ffff7c787c1 in  () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#70 0x00007ffff7b87a7c in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#71 0x00007ffff7b91ba9 in gst_pad_send_event () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#72 0x00007fff9817d11c in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#73 0x00007ffff7b866dc in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#74 0x00007ffff7b85ef0 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#75 0x00007ffff7b927bc in gst_pad_sticky_events_foreach () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#76 0x00007fff98188a2f in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#77 0x00007fff98186625 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#78 0x00007fff98186f2d in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#79 0x00007ffff28af249 in  () at /usr/lib/x86_64-linux-gnu/libffi.so.8
#80 0x00007ffff28ae629 in  () at /usr/lib/x86_64-linux-gnu/libffi.so.8
#81 0x00007ffff7ab73fd in g_cclosure_marshal_generic () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#82 0x00007ffff7ab68fa in g_closure_invoke () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#83 0x00007ffff7ac94b3 in  () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#84 0x00007ffff7acfc41 in g_signal_emit_valist () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#85 0x00007ffff7acfda3 in g_signal_emit () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#86 0x00007fff99128cb9 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstcoreelements.so
#87 0x00007ffff7b87a7c in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#88 0x00007ffff7b880fe in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#89 0x00007ffff7b885b8 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#90 0x00007ffff7b85ef0 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#91 0x00007ffff7b91530 in gst_pad_push_event () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#92 0x00007ffff7b91b12 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#93 0x00007ffff7b8db5e in gst_pad_forward () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#94 0x00007ffff7b8dca5 in gst_pad_event_default () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#95 0x00007ffff7b87a7c in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#96 0x00007ffff7b880fe in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#97 0x00007ffff7b885b8 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#98 0x00007ffff7b85ef0 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#99 0x00007ffff7b91530 in gst_pad_push_event () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#100 0x00007ffff7b91b12 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#101 0x00007ffff7b8db5e in gst_pad_forward () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#102 0x00007ffff7b8dca5 in gst_pad_event_default () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#103 0x00007fff9912a044 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstcoreelements.so
#104 0x00007ffff7b87a7c in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#105 0x00007ffff7b880fe in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#106 0x00007ffff7b885b8 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#107 0x00007ffff7b85ef0 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#108 0x00007ffff7b91530 in gst_pad_push_event () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#109 0x00007fff99122053 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstcoreelements.so
#110 0x00007ffff7bc01c7 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#111 0x00007ffff79edac4 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#112 0x00007ffff79ed1b1 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#113 0x00007ffff2af3590 in start_thread (arg=0x7fff83fff640) at pthread_create.c:463
#114 0x00007ffff2f9a223 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

AndKe commented 3 years ago

@jaxxzer - Is there anything else I can do to help to get this fixed?

jaxxzer commented 3 years ago

@AndKe Can you describe your video streaming setup?

What camera are you using
how is it connected
what's the video format?
What do the video settings in QGC look like?

AndKe commented 3 years ago

@jaxxzer This is the stock 3DR Gimbal setup with Gopro4 Black then the IMX6 sends H264 on port 5600 Video settings in QGC say simply "3DR Solo" - or - they may say "UDP h264 port 5600 (if I manually open the connection)

This is not about video format - as it will work just fine on a Nvidia equipped laptop or workstation, or QGC for Android. This is only affecting Intel graphics (most mid-end laptops with long battery life)

DonLakeFlyer commented 3 years ago

This crashing on Intel graphics drivers used to happen quite a bit prior to the changes by @andrewvoznytsa. The Sentera folks had lots of examples with their customers. If this has come back it's a significant problem and will affect a bunch of users.

@antiheavy FYI

jaxxzer commented 3 years ago

Can you upload a gstreamer debug log? Select the highest debug level from the Application Settings -> Console page, then restart the app and connect the video to capture the crash situation in the log.

kemen209 commented 3 years ago

I had similar problem:

I build QGC master branch with gstreamer 1.18.1. The QGC will crash on gstreamer init.

Stack trace:
1  gst_value_deserialize                     libgstreamer_1_0_0     0x439421b      
2  gst_value_deserialize                     libgstreamer_1_0_0     0x43944b6      
3  gst_structure_to_string                   libgstreamer_1_0_0     0x4371457      
4  gst_caps_from_string                      libgstreamer_1_0_0     0x431bbc6      
5  gst_static_caps_get                       libgstreamer_1_0_0     0x431bdbc      
6  gst_static_pad_template_get               libgstreamer_1_0_0     0x4351f95      
7  gst_element_class_add_static_pad_template libgstreamer_1_0_0     0x432d290      
8  libgsta52dec                                                     0x66341bb2     
9  g_type_class_ref                          libgobject_2_0_0       0x63a63167     
10 gst_element_register                      libgstreamer_1_0_0     0x4330f93      
11 gst_a52dec_get_type                       libgsta52dec           0x66342d5a     
12 gst_plugin_load_file                      gstreamer_1_0_0        0x7fffb98e1978 
13 gst_pipeline_use_clock                    gstreamer_1_0_0        0x7fffb98df43e 
14 gst_registry_scan_path                    gstreamer_1_0_0        0x7fffb98efcbc 
15 gst_registry_scan_path                    gstreamer_1_0_0        0x7fffb98efbd1 
16 gst_registry_scan_path                    gstreamer_1_0_0        0x7fffb98ef57f 
17 gst_update_registry                       gstreamer_1_0_0        0x7fffb98f03dd 
18 gst_query_writable_structure              gstreamer_1_0_0        0x7fffb98ed5e2 
19 gst_update_registry                       gstreamer_1_0_0        0x7fffb98efd77 
20 gst_version_string                        gstreamer_1_0_0        0x7fffb988adcc 
21 g_option_context_parse                    glib_2_0_0             0x7fffa0b0ac49 
22 gst_init_check                            gstreamer_1_0_0        0x7fffb988a52f 
23 GStreamer::initialize                     GStreamer.cc       198 0x7ff6a3017752 
24 QGCApplication::QGCApplication            QGCApplication.cc  357 0x7ff6a2d25805 
25 main                                      main.cc            342 0x7ff6a2e9884c 
26 invoke_main                               exe_common.inl     79  0x7ff6a3104479 
27 __scrt_common_main_seh                    exe_common.inl     288 0x7ff6a310435e 
28 __scrt_common_main                        exe_common.inl     331 0x7ff6a310421e 
29 mainCRTStartup                            exe_main.cpp       17  0x7ff6a3104509 
30 BaseThreadInitThunk                       KERNEL32               0x7ff80da27034 
31 RtlUserThreadStart                        ntdll                  0x7ff80eb9d0d1

And some exception in console:

Exception at 0x7ff80c46d759, code: 0xe06d7363: C++ exception, flags=0x1 (execution cannot be continued) (first chance) in libmfxhw64!MFXVideoVPP_GetVPPStat

Exception at 0x7ff80c46d759, code: 0xe06d7363: C++ exception, flags=0x1 (execution cannot be continued) (first chance) in libmfxhw64!MFXVideoVPP_GetVPPStat

Exception at 0x7ff80c46d759, code: 0xe06d7363: C++ exception, flags=0x1 (execution cannot be continued) (first chance) in libmfxhw64!MFXVideoVPP_GetVPPStat

Exception at 0x7ff80c46d759, code: 0xe06d7363: C++ exception, flags=0x1 (execution cannot be continued) (first chance) in libmfxhw64!MFXVideoVPP_GetVPPStat

Exception at 0x7ff80c46d759, code: 0xe06d7363: C++ exception, flags=0x1 (execution cannot be continued) (first chance) in libmfxhw64!MFXVideoVPP_GetVPPStat

Online search show that the exception related to Intel Media SDK. And i had two graphic card: Intel HD Graphics P530, and NVIDIA Quadro M1000M.

And i test it with gstreamer v1.18.2, v1.18.1, v1.14.4，only v1.14.4 working correctly. It's look like a bug in gstreamer v1.18.

AndKe commented 3 years ago

full log: log.zip

the final moment :

GStreamerLog: (NULL) created caps 0x7f288809a4f0
GStreamerLog: (NULL) created structure 0x7f288c57f9d0
GStreamerLog: (NULL) created caps features 0x55be237e5280
GStreamerLog: (NULL) trying field name 'format'
GStreamerLog: (NULL) trying type name 'string'
GStreamerLog: (NULL) Exceeding pre-allocated array
GStreamerLog: (NULL) trying field name 'width'
GStreamerLog: (NULL) trying type name 'int'
GStreamerLog: (NULL) trying field name 'height'
GStreamerLog: (NULL) trying type name 'int'
GStreamerLog: (NULL) trying field name 'framerate'
GStreamerLog: (NULL) trying type name 'fraction'
GStreamerLog: (NULL) trying field name 'interlace-mode'
GStreamerLog: (NULL) trying type name 'string'
GStreamerLog: <vaapidisplayglx0> 0x7f288c494100 ref 4->5
GStreamerLog: (NULL) size 64x64, format NV12, flags 0x00000000
GStreamerLog: <vaapidisplayglx0> 0x7f288c494100 ref 5->6
GStreamerLog: (NULL) surface (nil)
GStreamerLog: (NULL) format NV12, size 64x64
GStreamerLog: <vaapidisplayglx0> 0x7f288c494100 ref 6->7
GStreamerLog: (NULL) image (nil)
GStreamerLog: (NULL) 0x7f288c4c21d0 unref 1->0
GStreamerLog: (NULL) image (nil)
GStreamerLog: (NULL) replace 0x7f288c494100 vaapidisplayglx0 (7) with (nil) (NONE) (0)
GStreamerLog: <vaapidisplayglx0> 0x7f288c494100 unref 7->6
GStreamerLog: (NULL) format YV12, size 64x64
GStreamerLog: <vaapidisplayglx0> 0x7f288c494100 ref 6->7
GStreamerLog: (NULL) image (nil)
Segmentation fault (core dumped)
andre@Aranand:~/prog/qgroundcontrol/build/staging$ gstreamer-codec-install

AndKe commented 3 years ago

@kemen209 gstreamer 1.18.3 is out .. since you had that good flow on testing them, could you please try (or post some info on how you do it?)

DonLakeFlyer commented 3 years ago

@jaxxzer Should we move down to 1.14? That what you get on Linux anyway. Whereas all other OS are at 1.18. @LorenzMeier Could really use that help with video from Auterion.

LorenzMeier commented 3 years ago

@arnaudthiercelin We need to look into this.

AndKe commented 3 years ago

@DonLakeFlyer Debian based distros are on 1.18.x too. Another odd thing is that webcam software and other things like video players and browsers(?) may be using gstreamer too.. and I don't know one single program that fails and crashes on video on those computers. I did also use jevois (OpenCV thingy) recently, and that did not fail. What I am saying is that there may be a slim chance that QGC is (maybe) somehow doing something "differently"

LorenzMeier commented 3 years ago

It's certainly the pipeline configuration, not GStreamer itself and it can be fixed.

ArnaudThiercelin commented 3 years ago

@DonLakeFlyer We are looking already in this issue with GStreamer 1.18.2 vs 1.14. I'll keep you guys posted. Should have more tomorrow Zurich time.

DonLakeFlyer commented 3 years ago

Thanks guys.

AndKe commented 3 years ago

@ArnaudThiercelin Did you had a chance to look at this issue?

kemen209 commented 3 years ago

@AndKe Same exception for GStreamer 1.18.3

patrickelectric commented 3 years ago

Hi, can you try to change between the video decoders available on the video settings if possible ?

andrewvoznytsa commented 3 years ago

Hey guys, just curious why we don't try to update qmlglsink? It has been changed during last year - probably good time for update. And from what I see from the stack trace it could be related.

I tried to reproduce that problem on my side and nothing (the only one 9th gen which I've is MBP, rest are 4th gen and recent AMD systems - no luck with hardware) - yet another hard to debug case.

AndKe commented 3 years ago

@patrickelectric This is strange, I did not expect such results: Before connecting 3DR Solo, I chose "Force VA-API decoder" started QGC - video, working fine. So I chose "Force Nvidia" (this laptop does not have Nvidia GPU) strangely - after restarting QGC, it too worked fine. So I chose "Force software renderer" , restarted QGC - that too worked So I chose "default" again - and restarted , expecting crash on start.. and that too worked now..

Will try to reproduce on another computer on Monday, unless anyone can explain how/why all worked now..

patrickelectric commented 3 years ago

@AndKe, that's indeed really interesting, it may be a problem with the default values for the video decoder, or the settings system somehow. But let me know how it goes, I'll try to replicate it here.

patrickelectric commented 3 years ago

Hi @AndKe I tried to replicate under linux without success, I'll try on windows next week. @kemen209 can you test with the hardware decode video configuration ?

AndKe commented 3 years ago

@patrickelectric Now, I retried everything on today's QGC master in the i9-10900 CPU (Intel graphics) To my surprise, it crashes like before regardless of software/default setting, a bit different, but it does.

Default:

GStreamerAPILog: (NULL) created caps 0x7f3bec0d39e0
GStreamerAPILog: (NULL) created structure 0x7f3c54257e70
GStreamerAPILog: (NULL) created caps features 0x55d9eb38a040
GStreamerAPILog: (NULL) trying field name 'format'
GStreamerAPILog: (NULL) trying type name 'string'
GStreamerAPILog: (NULL) Exceeding pre-allocated array
GStreamerAPILog: (NULL) trying field name 'width'
GStreamerAPILog: (NULL) trying type name 'int'
GStreamerAPILog: (NULL) trying field name 'height'
GStreamerAPILog: (NULL) trying type name 'int'
GStreamerAPILog: (NULL) trying field name 'framerate'
GStreamerAPILog: (NULL) trying type name 'fraction'
GStreamerAPILog: (NULL) trying field name 'interlace-mode'
GStreamerAPILog: (NULL) trying type name 'string'
GStreamerAPILog: <vaapidisplayglx0> 0x7f3be89356f0 ref 2->3
GStreamerAPILog: (NULL) size 64x64, format NV12, flags 0x00000000
GStreamerAPILog: <vaapidisplayglx0> 0x7f3be89356f0 ref 3->4
GStreamerAPILog: (NULL) surface (nil)
GStreamerAPILog: (NULL) format NV12, size 64x64
GStreamerAPILog: <vaapidisplayglx0> 0x7f3be89356f0 ref 4->5
GStreamerAPILog: (NULL) image (nil)
GStreamerAPILog: (NULL) 0x7f3c54058340 unref 1->0
GStreamerAPILog: (NULL) image (nil)
GStreamerAPILog: (NULL) replace 0x7f3be89356f0 vaapidisplayglx0 (5) with (nil) (NONE) (0)
GStreamerAPILog: <vaapidisplayglx0> 0x7f3be89356f0 unref 5->4
GStreamerAPILog: (NULL) format YV12, size 64x64
GStreamerAPILog: <vaapidisplayglx0> 0x7f3be89356f0 ref 4->5
GStreamerAPILog: (NULL) image (nil)
Segmentation fault (core dumped)
andre@Aranand:~/prog/qgroundcontrol/build/staging$ 

software decoder:

GStreamerAPILog: <rtpjitterbuffer0:sink> 0x7f8dac06e810 ref 1->2
GStreamerAPILog: <rtpjitterbuffer0> 0x7f8dac14fba0 ref 7->8
GStreamerAPILog: <rtpjitterbuffer0:sink> calling chainfunction &gst_rtp_jitter_buffer_chain with buffer buffer: 0x7f8d446e6240, pts 0:00:02.862747689, dts 0:00:02.862747689, dur 99:99:99.999999999, size 1400, offset none, offset_end none, flags 0x0
GStreamerAPILog: (NULL) buffer 0x7f8d446e6240, idx 0, length 1, flags 0001
GStreamerAPILog: (NULL) buffer 0x7f8d446e6240, idx 0, length 1
GStreamerAPILog: (NULL) 0x7f8d446f1e30 ref 1->2
GStreamerAPILog: (NULL) lock 0x7f8d446f1e30: state 00010000, access_mode 1
GStreamerAPILog: (NULL) unlock 0x7f8d446f1e30: state 00010101, access_mode 1
GStreamerAPILog: (NULL) 0x7f8d446f1e30 unref 2->1
GStreamerAPILog: <rtpjitterbuffer0> Received packet #7592 at time 0:00:02.862747689, discont 0, rtx 0
GStreamerAPILog: (NULL) Locking from thread 0x55f91816a980
GStreamerAPILog: (NULL) Locked from thread 0x55f91816a980
GStreamerAPILog: <rtpjitterbuffer0> dtsdiff +0:00:00.001559215 rtptime +0:00:00.000000000, clock-rate 90000, diff +0:00:00.001559215, jitter: 0:00:00.009124356
GStreamerAPILog: <rtpjitterbuffer0> updated packet_rate: 723
GStreamerAPILog: <rtpjitterbuffer0> max_dropout: 43380, max_misorder: 1446
GStreamerAPILog: <rtpjitterbuffer0> expected #7592, got #7592, gap of 0
GStreamerAPILog: <rtpjitterbuffer0> Clearing gap packets
GStreamerAPILog: <rtpjitterbuffer0> Pushed packet #7592, now 237 packets, head: 0, percent -1
GStreamerAPILog: <rtpjitterbuffer0> waiting till 0:00:03.420827669 and earliest timeout is at 0:00:03.420827669
Segmentation fault (core dumped)

mavlink / qgroundcontrol