QGC doesn't display video stream

[adolgarev]

Hello,

I followed instructions in the README and tried t3.12.7 and t3.12.6 releases but QGC doesn't automatically display the video stream. I create ball-Fake source h264 rtp stream and can add it successfully manually in QGC settings, QGC then displays the stream as expected. But automatically this doesn't happen. I see that mavlink-camera-manager sends heartbeats and QGC reacts to them and requests MAV_CMD_REQUEST_CAMERA_INFORMATION. In mavlink-camera-manager logs

2024-03-10T04:22:38.616028Z TRACE           MavReceiver ThreadId(13) receiver_loop: src/mavlink/manager.rs:104: Message received: MavHeader { system_id: 255, component_id: 190, sequence: 11 }, COMMAND_LONG(COMMAND_LONG_DATA { param1: 1.0, param2: 0.0, param3: 0.0, param4: 0.0, param5: 0.0, param6: 0.0, param7: 0.0, command: MAV_CMD_REQUEST_CAMERA_INFORMATION, target_system: 1, target_component: 100, confirmation: 0 })

But mavlink-camera-manager neither acknowledges the message not sends the information back. I can verify this by watching messages on mavproxy to which mavlink-camera-manager is connected. As a result QGC reports "Giving up requesting camera info from 1 100"

Do you know what can be the problem? Thanks.

See the full log mavlink-camera-manager.2024-03-10-04.log attached

[joaoantoniocardoso]

Hi! Thanks for reporting.

I create ball-Fake source h264 rtp stream and can add it successfully

You mean, rtsp? I just submitted a PR fixing it.

[joaoantoniocardoso]

@adolgarev could you try the new version (t3.12.8)? Then we reopen if it is still not working.

Thanks!

[joaoantoniocardoso]

@adolgarev I released a new version (t3.12.9) because the previous fix wasn't enough for older GStreamer versions (like 1.23.4).

[adolgarev]

@joaoantoniocardoso thank you for your response. Recent build produces the same result, please find details below.

I configure the stream like below

I create ball-Fake source h264 rtp stream and can add it successfully You mean, rtsp? I just submitted a PR fixing it.

I believe it is RTP over UDP

If I add stream manually

It displays ok

But if I let QGC discover stream by itself I see

and no video

In logs mavlink-camera-manager sends only heartbeats, no message ack and no response with camera caps.

System id and component id looks right. I wonder why mavlink-camera-manager doesn't respond.

The new log: mavlink-camera-manager.2024-03-11-19.log

[joaoantoniocardoso]

Hi, I'm able to reproduce the issue, I'll investigate it.

[joaoantoniocardoso]

@adolgarev can you test #357 ?

• mavlink-camera-manager-aarch64

• mavlink-camera-manager-armv7

• mavlink-camera-manager-linux-desktop

[adolgarev]

It works! Thank yo very much.

There is one warning in QGC which is not fatal but maybe worth a look.

[joaoantoniocardoso]

@adolgarev from my tests, some unresponsiveness on the mavlink camera manager side makes the messages from the QGC be interpreted and answered too late, thus failing to start the stream.

We are still working on that.

[joaoantoniocardoso]

@adolgarev from my tests, some unresponsiveness on the mavlink camera manager side makes the messages from the QGC be interpreted and answered too late, thus failing to start the stream.

We are still working on that.

this PR solves this.

@adolgarev, the Unable to load camera parameters from camera definition is something that I saw during the development of this fix, but I'm not seeing it anymore, so I believe it was related to the performance issue.

[adolgarev]

Thanks! I'll give it a try

[adolgarev]

Can you please make a build with the PR above? Thanks

[joaoantoniocardoso]

@adolgarev Now the changes are in master:

mavlink-camera-manager-aarch64 mavlink-camera-manager-armv7 mavlink-camera-manager-linux-desktop

[adolgarev]

I still see CameraControlLog: Unable to load camera parameters from camera definition on QGC, I believe it is because I'm using fake source.

[patrickelectric]

@adolgarev but the video is showing ? Or only the controls ?

[DmitriyLugovoy]

hello! I modified src/stream/pipeline/v4l_pipeline.rs::76 VideoEncodeType::Yuyv => { format!( concat!( "v4l2src device={device} do-timestamp=true", " ! videoconvert", " ! x264enc", " ! h264parse", " ! capsfilter name={filter_name} caps=video/x-h264,stream-format=avc,alignment=au,width={width},height={height},framerate={interval_denominator}/{interval_numerator}", " ! tee name={video_tee_name} allow-not-linked=true", " ! rtph264pay aggregate-mode=zero-latency config-interval=10 pt=96", " ! tee name={rtp_tee_name} allow-not-linked=true" ), and src/stream/sink/image_sink.rs::355 VideoEncodeType::Yuyv => { // For h264, we need to filter-out unwanted non-key frames here, before decoding it. let filter = gst::ElementFactory::make("identity") .property("drop-buffer-flags", gst::BufferFlags::DELTA_UNIT) .property("sync", false) .build()?; let decoder = gst::ElementFactory::make("avdec_h264") .property_from_str("lowres", "2") // (0) is 'full'; (1) is '1/2-size'; (2) is '1/4-size' .build()?; decoder.has_property("discard-corrupted-frames", None).then(|| decoder.set_property("discard-corrupted-frames", true)); _transcoding_elements.push(filter); _transcoding_elements.push(decoder); } Now I'm able to stream from YUYV to QGC.

Also I updated src/mavlink/mavlink_camera.rs::279 flags: mavlink::common::CameraCapFlags::CAMERA_CAP_FLAGS_HAS_VIDEO_STREAM | mavlink::common::CameraCapFlags::CAMERA_CAP_FLAGS_CAN_CAPTURE_IMAGE_IN_VIDEO_MODE | mavlink::common::CameraCapFlags::CAMERA_CAP_FLAGS_HAS_BASIC_ZOOM | mavlink::common::CameraCapFlags::CAMERA_CAP_FLAGS_HAS_BASIC_FOCUS,

what I need to do to see zoom and focus controls in QGC? Will appreciate for your help!

[DmitriyLugovoy]

here logs of QGC and camera_def.xml and camera-manager logs QGC_logs.txt HP HD Camera: HP HD Camera_HP HD Camera: HP HD Camera_000.zip mavlink-camera-manager.2024-03-22-09.log

[joaoantoniocardoso]

hello! I modified src/stream/pipeline/v4l_pipeline.rs::76 VideoEncodeType::Yuyv => { format!( concat!( "v4l2src device={device} do-timestamp=true", " ! videoconvert", " ! x264enc", " ! h264parse", " ! capsfilter name={filter_name} caps=video/x-h264,stream-format=avc,alignment=au,width={width},height={height},framerate={interval_denominator}/{interval_numerator}", " ! tee name={video_tee_name} allow-not-linked=true", " ! rtph264pay aggregate-mode=zero-latency config-interval=10 pt=96", " ! tee name={rtp_tee_name} allow-not-linked=true" ), and src/stream/sink/image_sink.rs::355 VideoEncodeType::Yuyv => { // For h264, we need to filter-out unwanted non-key frames here, before decoding it. let filter = gst::ElementFactory::make("identity") .property("drop-buffer-flags", gst::BufferFlags::DELTA_UNIT) .property("sync", false) .build()?; let decoder = gst::ElementFactory::make("avdec_h264") .property_from_str("lowres", "2") // (0) is 'full'; (1) is '1/2-size'; (2) is '1/4-size' .build()?; decoder.has_property("discard-corrupted-frames", None).then(|| decoder.set_property("discard-corrupted-frames", true)); _transcoding_elements.push(filter); _transcoding_elements.push(decoder); } Now I'm able to stream from YUYV to QGC.

Also I updated src/mavlink/mavlink_camera.rs::279 flags: mavlink::common::CameraCapFlags::CAMERA_CAP_FLAGS_HAS_VIDEO_STREAM | mavlink::common::CameraCapFlags::CAMERA_CAP_FLAGS_CAN_CAPTURE_IMAGE_IN_VIDEO_MODE | mavlink::common::CameraCapFlags::CAMERA_CAP_FLAGS_HAS_BASIC_ZOOM | mavlink::common::CameraCapFlags::CAMERA_CAP_FLAGS_HAS_BASIC_FOCUS,

what I need to do to see zoom and focus controls in QGC? Will appreciate for your help!

Cool, I'll check your logs soon, but can you make a draft PR with your changes?

The zoom should show up if it is shown on the front-end page hosted by MCM.. as in, it will be shown if the camera has the zoom as a UCV/V4L2 control. (it also is possible that we are doing something wrong and not detecting that control for that camera somehow..)

[DmitriyLugovoy]

ERROR: Permission to mavlink/mavlink-camera-manager.git denied to DmitriyLugovoy.

[joaoantoniocardoso]

ERROR: Permission to mavlink/mavlink-camera-manager.git denied to DmitriyLugovoy.

You need to fork the project, sorry =)

[adolgarev]

@adolgarev but the video is showing ? Or only the controls ?

Yes, video is showing

[joaoantoniocardoso]

@adolgarev but the video is showing ? Or only the controls ?

Yes, video is showing

@adolgarev good, so I won't bother looking into that. Feel free to create new issues if you find anything else, or if you want to discuss new features, thanks!

[DmitriyLugovoy]

@joaoantoniocardoso https://github.com/mavlink/mavlink-camera-manager/pull/369/files

[tranhien1612]

@adolgarev @joaoantoniocardoso i will take you the steps i did to display the video on QGC. I don't know if it's true or not, so I would like to have steps to accomplish that task. I tried with some my step below but it`s not work.

1. sudo ./mavlink-camera-manager --mavlink=tcpout:0.0.0.0:14444 --verbose

2. I config "Camera: Fake Source" and it had some problems.

Please help me to be able to watch videos via QGC! Thanks!

[joaoantoniocardoso]

Hi @adolgarev, can you post the output of the following terminal command?

gst-launch-1.0 --version

We strongly advise to use the newest GStreamer (some after 1.24), but 1.22 should not give that error.

[tranhien1612]

@joaoantoniocardoso I checked my gst version as below, maybe the error is caused by low version.

gst-launch-1.0 version 1.20.1
GStreamer 1.16.3
https://launchpad.net/distros/ubuntu/+source/gstreamer1.0

[joaoantoniocardoso]

@adolgarev Probably yes. I don't plan to support old GStreamer, so let me know if you need help getting the current version for your platform.

Also, I should add a check in the code and add this information to the readme.

[tranhien1612]

@joaoantoniocardoso I am trying to display image from WebCam to WebRTC Development UI. I see it doesn't display the camera and video information in web, is this related to the old version of GStreamer?

[tranhien1612]

@joaoantoniocardoso I have installed gst according to the link: gsteamer-1.22 , after installation the version of gst is 1.22 as shown below.

gst-launch-1.0 version 1.22.9
GStreamer 1.22.9
Unknown package origin

then i rebuilt the source code and tried again:

[joaoantoniocardoso]

@joaoantoniocardoso I am trying to display image from WebCam to WebRTC Development UI. I see it doesn't display the camera and video information in web, is this related to the old version of GStreamer?

Hi! The WebRTC only works with h264, not YUYV, and we don't support encoding yet.

then i rebuilt the source code and tried again:

Good, that version should be enough, although 1.22 can have some thread leaks on the WebRTC, depending on the platform and OS, the safest is 1.24.

The error is because in mentioned guide, it is not building most of the plugins. Here is the config I use:

meson build\
    --buildtype=release\
    --strip\
    -D bad=enabled\
    -D devtools=enabled\
    -D doc=disabled\
    -D ges=disabled\
    -D gpl=enabled\
    -D gst-plugins-bad:openh264=disabled\
    -D gst-plugins-bad:rtp=enabled\
    -D gst-plugins-base:tcp=enabled\
    -D gst-plugins-good:cairo=disabled\
    -D gst-plugins-good:jpeg=enabled\
    -D gst-plugins-good:rtsp=enabled\
    -D gst-plugins-good:udp=enabled\
    -D gst-plugins-good:v4l2=enabled\
    -D gst-plugins-good:vpx=enabled\
    -D gst-plugins-ugly:x264=enabled\
    -D gst-rtsp-server:examples=enabled\
    -D introspection=disabled\
    -D libav=enabled\
    -D nls=disabled\
    -D orc=disabled\
    -D python=disabled\
    -D qt5=disabled\
    -D rs=disabled\
    -D rtsp_server=enabled\
    -D tests=disabled\
    -D tls=enabled\
    -D ugly=enabled

Thanks!