Frame rate dropping when FLIR Blackfly-s is synced with a LiDAR

[kavishmshah]

Hi all,

I'm using a BlackFly-S USB camera and have it synced with an Ouster LiDAR using hardware sync. The camera receives pulses at a frequency of 10 Hz from the LiDAR. I've set the below parameters for the sync:

'blackfly_s': {
    'debug': False,
    'compute_brightness': False,
    'adjust_timestamp': True,
    'dump_node_map': False,
    'gain_auto': 'Continuous',
    'pixel_format': 'BayerRG8',
    'exposure_auto': 'Continuous',
    'user_set_selector': 'UserSet0',
    'user_set_load': 'Yes',
    'frame_rate_auto': 'Off',
    'frame_rate': 40.0,
    'frame_rate_enable': True,
    'buffer_queue_size': 10,
    'trigger_mode': 'On',
    'trigger_source': 'Line3',
    'line2_selector': 'Line2',
    'trigger_activation': 'RisingEdge',
    'chunk_mode_active': True,
    'chunk_selector_frame_id': 'FrameID',
    'chunk_enable_frame_id': True,
    'chunk_selector_exposure_time': 'ExposureTime',
    'chunk_enable_exposure_time': True,
    'chunk_selector_gain': 'Gain',
    'chunk_enable_gain': True,
    'chunk_selector_timestamp': 'Timestamp',
    'chunk_enable_timestamp': True,

}

With Spinview, I see: Camera FPS: 10Hz, but when I run it with ROS, and measure frame rate with ros2 topic hz /flir_camera/image_raw, the frame rate is around 8.5. Additionally, with 'pixel_format': 'BGR8', frame rates are as low as 7 FPS. Is there any workaround this?

Thanks! Kavish

[berndpfrommer]

I believe this to be a "feature" of ros2 topic hz. It consumes so much CPU that it drops frames, hence under reports hz. Can you verify this by running "top" to see how much CPU it uses?

[kavishmshah]

Thanks for your reply. Below is the output of 'top'

[berndpfrommer]

Are you running "ros2 topic hz" at this point?

[kavishmshah]

Nope

[berndpfrommer]

The purpose of running "top" was to see how much CPU the "ros2 topic hz" command consumes...

[kavishmshah]

My bad, now I have "ros2 topic hz" running.

[berndpfrommer]

Hmm, 7%. That does not look like it's the problem. Could the RMW be dropping images? Not sure how to debug this. Does not look like a driver problem to me. Can you record into a rosbag and see how many images you get, then divide by time to get the frame rate?

[kavishmshah]

Left terminal window: So I wrote a script which calculates the number messages, duration of between first and last messages along with the frequency as seen below.

Right terminal window: frequency based on "ros2 topic hz"

[kavishmshah]

And below is the output with Pixel Format: BGR8. Significantly lesser frames.

[berndpfrommer]

OK, so the driver works fine in Bayer mode. What camera are you actually using (exact model)? What resolution are you running? Reason I'm asking is to see if you are close to saturating bandwidth. Then if you run RGB, that's a factor 3 more data than Bayer. Also I see you have auto exposure turned on. Not sure what happens when the image gets too dark. The frame rate may drop lower than 10Hz.

[kavishmshah]

Yes the driver works fine in Bayer mode and I run another debayer node to convert it to RGB which also works fine with the required 10 FPS with my hardware sync. For the resolution, I'm running the default resolution without changing anything in the ROS2 driver. For the auto exposure, I haven't tried out much with extremely dark environments. WIth the ROS1(Noetic) driver and fairly low-lit conditions, it was working fine.

Another interesting find was when running the same ROS2 driver with the same configs on a NVIDIA devkit and could see the frame rates drop even though spinview showed 10 FPS in sync mode. Without the debayer node, FPS was ~8.5 and with debayer node running FPS was ~7.5. And with htop, RAM and CPU cores were barely filled up.

[berndpfrommer]

I don't see any issues on my NVidia Xavier AGX using a stereo USB3 Blackfly S configuration, but then those cameras are just 1.5 megapixel. Without knowing the camera resolution I can only speculate as to why you are seeing low FPS on your NVidia system. On the Jetson TX USB performance was much weaker than on comparable Intel systems. Also the Jetson TX did not like memory copies. That said, I'd be surprised if the ROS2 driver (without any subscribers) showed lower FPS than SpinView running on the same host. If you don't subscribe, the ROS2 driver just counts the frames without memory access.

[kavishmshah]

So the camera is BFS-U3-20S4C-C: 2.0 MP, 175 FPS, Sony IMX422 Color, C-Mount. Resolution is 1240x1616. I will run this again on the devkit to double check my findingsand share the results with and without the debayer node running.

[berndpfrommer]

Strange. That's a fairly low resolution these days. You should be able to pull the full 175 FPS (unsynced). I assume you don't have anything else running on your system that is memory intensive, like the LIDAR node.

[kavishmshah]

Nope nothing really, I'm just syncing the LiDAR's pulses with the camera but not running the LiDAR's ROS2 node. It has a 64GB RAM of which just around 8GB was used up. With Spinview it runs smoothly. I will probably double check the frame rates with and without the sync with both Spinview and ROS2 driver and report back what I find.

[berndpfrommer]

Curious to hear what you find. Just to clarify: low memory consumption does not mean low memory bandwidth utilization. You can bog a system down by just copying data around. Usually the process doing the memory copying will show high CPU utilization, but the rest of the system will suffer as well.

[berndpfrommer]

@kavishmshah Any news here?

[kavishmshah]

@berndpfrommer Again apologies for the delayed response. The bandwidth related issue on the Devkit and PC was solved by the following:

1. Changing the DDS to cyclone and tuning it with multicast and localhost settings.
2. In addition to this, we also increased the memory buffer size.
3. Setting MTU to 9000 also helped.

We tested the BlackFly S USB and gigE cameras for this. Got it working in sync and non-sync modes with a LiDAR. I occasionally see the debayer node crash though. I haven't been able to replicate it and it seems to be random. I'm replicating this on another system and shall put the steps which resolved this issue very soon.

Thanks!