search

christianrauch / camera_ros

ROS 2 node for libcamera supported cameras (V4L2, Raspberry Pi Camera Modules)
https://libcamera.org
MIT License
68 stars 30 forks source link

IMX477 sensor configured as a sink #37

Closed christian-nils closed 7 months ago

christian-nils commented 7 months ago

Hi @christianrauch,

We have two IMX477 HQ cameras connected to a Raspberry Pi 5. One is configured as a sink while the other one is configured as a source. Using rpicam-vid I can receive frames at 30FPS from both cameras but I am interested in using your implementation as it solves issues we have today with h264-encoded streams. Below are the commands I use with rpicam-vid

For the sink:

LIBCAMERA_RPI_CONFIG_FILE=/etc/rpi_apps.yaml && rpicam-vid -t 0 -n --camera 0 --codec libav --libav-format h264 --libav-video-codec libx264 --libav-video-codec-opts "tune=zerolatency;preset=ultrafast;g=30" --width 2028 --height 1080 --denoise off --lens-position 0.0 --hdr off --framerate 30 -o udp://127.0.0.1:5555

For the source:

rpicam-vid -t 0 -n --camera 1 --codec libav --libav-format h264 --libav-video-codec libx264 --libav-video-codec-opts "tune=zerolatency;preset=ultrafast;g=30" --width 2028 --height 1080 --denoise off --lens-position 0.0 --hdr off --framerate 30 -o udp://127.0.0.1:5556

Using camera_ros with Raspberry Pi's libcamera, I can generate ROS2 compressed images, which is awesome but with some limitations. It only works with low FPS. For instance at 10 FPS, the following commands are used and works:

For the sink:

LIBCAMERA_RPI_CONFIG_FILE=/etc/rpi_apps.yaml && ros2 run  camera_ros camera_node --ros-args -p camera:=0 -p width:=2028 -p height:=1080 -p role:=still -r /camera/image_raw:=/left_cam/raw -r /camera/image_raw/compressed:=/left_cam/compressed -r /camera/camera_info:=/left_cam/camera_info -p FrameDurationLimits:="[100000,100000]" -p AeFlickerMode:="0" -p AeEnable:="false" -p HdrMode:="0" -p NoiseReductionMode:="0" -p format:="YUYV"

For the source:

ros2 run  camera_ros camera_node --ros-args -p camera:=1 -p width:=2028 -p height:=1080 -p role:=still -r /camera/image_raw:=/right_cam/raw -r /camera/image_raw/compressed:=/right_cam/compressed -r /camera/camera_info:=/right_cam/camera_info -p FrameDurationLimits:="[100000,100000]" -p AeFlickerMode:="0" -p AeEnable:="false" -p HdrMode:="0" -p NoiseReductionMode:="0" -p format:="YUYV"

I tried at 20 FPS, and it worked sometimes. I realized that it worked when the ambient brightness was significantly higher than in the no-working cases. So I think it has to do with the exposure time. I cannot fix the exposure time as I have the same problem as in #33. My gut feeling is that the program does not handle correctly when an external trigger is received while the camera is still under exposure, I can get few first frames with visible first rows, but then it loops over those frames forever (the publishing rate is correct, ~20 Hz).

image

Is it a misconfiguration on my side or does camera_ros not support such setup?

christian-nils commented 7 months ago

Alright, think I found the fix, I made a PR: https://github.com/christianrauch/camera_ros/pull/39 Inspired by https://github.com/raspberrypi/rpicam-apps/blob/414a7383464b98f21f5e5381a16cc73ae0350ba6/core/rpicam_app.cpp#L757C15-L757C32.

@christianrauch: I hope this is useful. Let me know if any changes are needed for the PR.