search

ArduPilot / ardupilot_gazebo

Plugins and models for vehicle simulation in Gazebo Sim with ArduPilot SITL controllers
GNU Lesser General Public License v3.0
97 stars 91 forks source link

Video update rate and cuda are not working. #108

Open IuliuNovac opened 3 months ago

IuliuNovac commented 3 months ago

Bug report

Issue details

I am changing the camera settings for the simulation but enabling cuda and changing update rate are not working. When enabeling cuda, true, there is no more video output. When changing the 20, there is no change in FPS. The image resolution changes without issues.

I am using an NVIDIA L4 gpu, with NVIDIA L4 VRAM. There is barely any usage.

This is the configuration file.

  <sensor name="camera" type="camera">
    <pose>0 0 0 -1.57 -1.57 0</pose>
    <camera>
      <horizontal_fov>2.0</horizontal_fov>
      <image>
        <width>32</width>
        <height>24</height>
      </image>
      <clip>
        <near>0.05</near>
        <far>15000</far>
      </clip>
    </camera>
    <always_on>1</always_on>
    <update_rate>20</update_rate>
    <visualize>1</visualize>

    <plugin filename="CameraZoomPlugin"
        name="CameraZoomPlugin">
      <max_zoom>125.0</max_zoom>
      <slew_rate>0.42514285714</slew_rate>
    </plugin>

    <plugin name="GstCameraPlugin"
        filename="GstCameraPlugin">
      <udp_host>127.0.0.1</udp_host>
      <udp_port>5600</udp_port>
      <use_basic_pipeline>true</use_basic_pipeline>
      <use_cuda>true</use_cuda>
    </plugin>

  </sensor>

Gsteamer command 

gst-launch-1.0 -v udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' ! rtph264depay ! avdec_h264 ! videoconvert ! fpsdisplaysink video-sink=autovideosink sync=false
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Got context from element 'autovideosink0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage/GstGLImageSink:sink: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage: sync = false
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
/GstPipeline:pipeline0/GstUDPSrc:udpsrc0.GstPad:src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
New clock: GstSystemClock
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264

Logs

[Msg] GstCameraPlugin: image topic [world/iris_runway/model/iris_with_gimbal/model/gimbal/link/pitch_link/sensor/camera/image]
[Msg] GstCameraPlugin: enable topic [world/iris_runway/model/iris_with_gimbal/model/gimbal/link/pitch_link/sensor/camera/image/enable_streaming]
[Dbg] [CameraSensor.cc:480] Enabling camera sensor: 'iris_with_gimbal::gimbal::pitch_link::camera' data generation.
[Msg] GstCameraPlugin:: streaming: started
[Dbg] [GstCameraPlugin.cc:407] GstCameraPlugin: creating generic pipeline
[Dbg] [GstCameraPlugin.cc:477] Using Cuda
[Msg] GstCameraPlugin: GStreamer element set state returned: 2
[Msg] GstCameraPlugin: starting GStreamer main loop

Version Master branch

Platform [ ] All [ ] AntennaTracker [X] Copter [ ] Plane [ ] Rover [ ] Submarine

IuliuNovac commented 3 months ago

The solution to increase the fps is running with higher frequency with -z 1000.

The cuda pipeline seems to be odd. Could someone explain how to set the right pipeline for cuda?

GST_DEBUG=3 gst-launch-1.0 -v udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' ! rtph264depay ! h264parse ! identity silent=false ! avdec_h264 ! videoconvert ! nvh264enc bitrate=800 preset=1 ! fpsdisplaysink video-sink=autovideosink sync=false

0:00:03.143219509 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.152016330 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.161007796 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.169893667 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.178655136 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.186627007 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.195297441 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.205585473 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.214132605 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.222478166 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.240257989 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.251110211 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.259886638 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.268951976 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.277225666 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.285682907 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.295426409 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.305170705 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.314307800 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.323525971 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.332682051 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.341781994 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.350402596 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.359266195 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.368698366 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.377041382 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.385405786 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
^Chandling interrupt.
0:00:03.394794017 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
prodrone-liu commented 2 months ago

I am also running the same command with use_cuda set to true. But GstCameraPlugin failed in my case. May I ask about your configurations of cuda? Logs:

[Msg] GstCameraPlugin:: streaming: started
[Dbg] [GstCameraPlugin.cc:407] GstCameraPlugin: creating generic pipeline
[Dbg] [GstCameraPlugin.cc:477] Using Cuda
(gz sim server:8009): GLib-GObject-CRITICAL **: 11:26:59.773: g_object_set: assertion 'G_IS_OBJECT (object)' failed
[Err] [GstCameraPlugin.cc:419] GstCameraPlugin: failed to create GStreamer elements
[Msg] GstCameraPlugin: starting GStreamer main loop