search

mavlink / qgroundcontrol

Cross-platform ground control station for drones (Android, iOS, Mac OS, Linux, Windows)
http://qgroundcontrol.io
3.3k stars 3.63k forks source link

Latency in QGC when Receiving H.265 Video Stream #12020

Open nechtobolshee opened 3 weeks ago

nechtobolshee commented 3 weeks ago

We are experiencing a consistent 150+ms latency when receiving H.265 video streams in QGroundControl (QGC). The same stream, when tested outside of QGC (using tools like gst-launch), does not show this level of latency, which suggests that the delay is occurring within QGC's internal handling of the stream.

Expected Behavior

Latency less than 50ms when processing QGC images (including decoding).

Current Behavior

Latency over 150ms when receiving H.265 video stream (150+ms latency inside QGC -> decoding and processing)

Steps to Reproduce:

Please provide an unambiguous set of steps to reproduce the current behavior

  1. Stream H.265 encoded video to QGC
  2. Observe the latency between the actual event and the video display in QGC
  3. Perform the same test using gst-launch or similar external tools to verify the stream performance outside of QGC (latency is much lower or non-existent).

System Information

Detailed Description

It is necessary to understand why this latency occurs. The low latency mode is enabled in the settings.

Could you please point me to the code area in QGC where I can see the final GStreamer pipeline used to receive the stream?

nechtobolshee commented 3 weeks ago

@HTRamsey I saw that you made a large update for GStreamer yesterday. Could you check this case? Also, where can I see the final GStreamer pipeline used to obtain the stream? Thanks.

HTRamsey commented 3 weeks ago

working on it now

HTRamsey commented 2 weeks ago

@nechtobolshee is this using the prebuilt daily or are you building yourself? I'm sure most of the latency is due to QGC itself but I'd be curious if it was any better if you were able to use gstreamer 1.24

nechtobolshee commented 2 weeks ago

@HTRamsey I'm building the master branch. Yes, you are right; the high latency is coming from QGC. As I mentioned, I have tried receiving the stream in two ways: using a native GStreamer pipeline (latency around 50 ms) and through QGC (latency increases to 180-220 ms). I'll try using GStreamer 1.24, but I don't think it will make a significant difference.

HTRamsey commented 2 weeks ago

Part of the problem might be that the ubuntu version uses an old backported plugin from early 1.22 (I think) to get it to work on 1.20/1.18/1.16. What I was thinking was that the newer plugin from 1.24 would work much better, but I think I have to make a couple small changes to get it to use gstreamer's provided plugin and not the QGC backported plugin

nechtobolshee commented 2 weeks ago

@HTRamsey Could you please provide information regarding the following points (I'm currently working on latency):

  1. Where can I find the configured GStreamer pipeline in QGC? I saw that QGC contains a dynamic builder for the GStreamer pipeline, but how can I get the resulting pipeline configuration? Specifically, I'm looking for the complete gst-launch-1.0 udpsrc ... string.
  2. Is it possible to retrieve the RTP timestamp (in PTS format) for each frame? If yes, could you direct me to the relevant code or part of the project?
HTRamsey commented 2 weeks ago

Everything you need will be in VideoManager/GStreamer. With all of the pipeline stuff being in the file GstVideoReceiver. My first guess would be that including the x265 plugin would help before messing with anything else. There's also a file in the CMake/find_modules folder that controls how to link against GStreamer.