MPEG-TS H.264 stream is not shown

ddatsko commented 9 months ago

MPEG-TS H.264 stream is not shown in QGC, however the video can be recorded well

Expected Behavior

When video source is set to MPEG-TS (h.264) and UDP port is set to 8081, the video should be shown in QGC after running gst-launch-1.0 videotestsrc ! video/x-raw, format=NV12, width=1920, height=1080 ! x264enc ! h264parse config-interval=-1 ! mpegtsmux ! udpsink host=127.0.0.1 port=8081

Current Behavior

When performing the steps above, the video is not shown in QGC. However, when the stream is running, the "Record" button becomes active and is fully functional. After recording the video, the .mkv file with recording contains the valid received video

Steps to Reproduce:

In QGC, set up video source to MPEG-TS (h.264) and UDP port to 8081
Run the gstreamer pipeline from above

System Information

Operating System: Ubuntu 22.04
QGC Version:
- v4.3.0 release from github
- Build from source in current master

Log Files and Screenshots

Log of QGroundControl launched when the stream is already running: QGC_log.txt

ddatsko commented 9 months ago

UPD: I was debugging the issue more deeply, and it looks that the problem is in the decodebin3 gstreamer plugin. When the decoder is set to decodebin instead of decodebin3 in the code, the video is played and I obtain the following pipeline inside of the QGC with properly selected avdec_h264 decoder: pipeline-with-new-decoder-pad However, if I keep the decodebin3, the pipeline is created only till this stage and looks like decodebin3 cannot properly handle the stream pipeline-with-decoder All this also fits well with the observation that video recording works well: there is just no decoding there and the stream goes directly into matroskamux and the file

After this, I was able to replicate the same problem with just gstreamer. When sending the MPEG-TS stream with the pipeline above, all these pipelines work for receiving: gst-launch-1.0 udpsrc port=8081 ! parsebin ! decodebin ! videoconvert ! autovideosink sync=False gst-launch-1.0 udpsrc port=8081 ! decodebin ! videoconvert ! autovideosink sync=False gst-launch-1.0 udpsrc port=8081 ! decodebin3 ! videoconvert ! autovideosink sync=False

However, the one created in QGC doesn't work: gst-launch-1.0 udpsrc port=8081 ! parsebin ! decodebin3 ! videoconvert ! autovideosink sync=False fails with Internal data stream error

I see two possible workarounds in this case:

Replace decodebin3 with decodebin, possibly loosing some functionality of it. This doesn't sound like a good option as decodebin3 is newer, promises to fix some weird behaviour of decodebin and has better dynamical connections switching
Remove parsebin and leave only decodebin3 in the pipeline. This sound like a valid option considering that decodebin3 should create a parsebin element inside (at least from what I can see from the image below, generated after running the last of three working pipelines described above). This, however, would require a second parser for recording into a file

fredowski commented 8 months ago

I dived into this 100m deep tank of shit of the gstreamer video stuff some time ago: https://github.com/mavlink/qgroundcontrol/pull/10420

My result is that it is impossible to setup a automagic pipeline that works for all video formats on all platforms. So if you figure out one pipeline setup that works for your particular video on your platform then you have to make sure that this works on all other platforms: linux, MacOS, Windows, Android. Then you have to make sure that you have not broken other video formats.

ericjohnson97 commented 8 months ago

I wanted to +1 this issue. I am seeing the same error in my QGC log on Windows when trying to receive a UDP h264 stream

[!] at D:\a\qgroundcontrol\qgroundcontrol\src\VideoReceiver\GstVideoReceiver.cc:1305 - "GStreamer error: Internal data stream error."

mavlink / qgroundcontrol