Progressively slowing down FPS

[ruthvik92]

OS : Ubuntu 22.04 ROS2: Humble Camera: iDS GV 5260CP C-HQ Rev2 ENSENSO_SDK_VERSION=3.5.1419

I am trying to see what's the maximum FPS that I can get from this camera. When plugged into ids_peak or nxView I see that my setup is able to achieve 40FPS. So, I modified your request_data_mono python script to continuously publish in a while loop. FPS starts at about 10 to 12 and dips down to 3 after some time. Looks like the time to complete the goal is increasing..

Is this the expected behaviour ?

#!/usr/bin/env python3
import sys
import time

import ensenso_camera.ros2 as ros2py

RequestDataMono = ros2py.import_action("ensenso_camera_msgs", "RequestDataMono")

def _main(node_name):
    node = ros2py.create_node(node_name, args=sys.argv)

    namespace = ros2py.get_param(node, "namespace", "")
    timeout = ros2py.get_param(node, "timeout", 60)

    goal = RequestDataMono.Goal()
    goal.parameter_set = ros2py.get_param(node, "parameter_set", "")

    goal.request_raw_images = ros2py.get_param(node, "raw_images", True)
    goal.request_rectified_images = ros2py.get_param(node, "rectified_images", True)

    goal.publish_results = True

    request_data_client_name = "request_data" if namespace == "" else namespace.rstrip("/") + "/request_data"
    request_data_client = ros2py.create_action_client(node, request_data_client_name, RequestDataMono)

    ros2py.wait_for_server(node, request_data_client, timeout_sec=timeout)

    while True:
        t1 = time.time()
        response = ros2py.send_action_goal(node, request_data_client, goal)
        if not response.successful():
            node.get_logger().warn("Action was not successful.")
        else:
            result = response.get_result()
            node.get_logger().info("Time:{}".format(time.time() - t1))
            if result.error.code != 0:
                node.get_logger().error(ros2py.format_error(result.error))

def main():
    ros2py.wrap_main_function(_main, "ensenso_camera_request_data_mono")

if __name__ == "__main__":
    main()

[benthie]

Hi @ruthvik92,

I have actually encountered the same problem once while porting our driver from ROS1 to ROS2. But soon after seeing the problem I changed to another computer and did some further improvements on the code; the problem has not occurred ever since. If you are working with the latest version of the driver, then I suspect it to be a hardware / setup problem.

To be sure, I just tested the FPS of an IDS GV-5040CP-M camera both on ROS1 and ROS2 and the results are the same and constant. In NxView the frame rate was beyond 40 FPS (and I also tested with binning=2 which further increases the frame rate), however, on ROS it has its limit at around 22 FPS, which is probably due to the message overhead generated by ROS.

For further investigation of the problem it would be very helpful if you could provide us with some logs: firstly the ROS log of the request_data_mono script as you've done before and secondly an NxLib log. For this you have to open a TCP port on the NxLib that is contained within the ROS camera node via the tcp_port parameter to the node (see the launch file example).

ros2 run ensenso_camera ensenso_camera_mono_node --ros-args -p serial:=1234! -p tcp_port:=0

Before you start requesting data via the script you then open NxTreeEdit and connect to the NxLib instance. In the upcoming window you then see a Debug Tab in which you can start/stop the logging.

From another internal support ticket I noticed that you are having MTU=1500. For best network performance we recommend setting the value to 9000 to enable Jumbo Frames.

Kind Regards Benny

[ruthvik92]

Sorry for the late response.. here are the log files for the node, client, and the nxLib. Note that I couldn't keep the nxLib logger running as long as the node/client because the size was growing rapidly. Let me know if you need one that is longer, I have a 1.4Gb file also...

also, how do i modify the value of MTU ?

[benthie]

I will have a look into the logs.

Regarding the MTU see this post.

[benthie]

Hi @ruthvik92,

I just evaluated the data you provided.

From the logs it can be seen that the problem is not on the NxLib side, where capture duration does not progressively increase.

The problem is on the ROS side, where the times between two accepted goals do increase.

To further investigate the problem I have a few questions:

1. Which version of our ros_driver do you use?
2. How is your camera connected to your PC? Directly or via one/several switches?
3. Did you see the slowing down also with the original request_data_mono script? Because in the version you posted above, the execute_at_rate method from our ROS2 compatibility layer is not used, which might cause some problems.
4. Does the problem still occur if you set goal.publish_results = False in the script?
5. I tested on Ubuntu 20.04 with Galactic and rmw_cyclonedds_cpp while you are using Ubuntu 22.04 with rmw_fastrtps_cpp. Could you try installing and using Cyclone DDS as explained here?
6. Regarding to this post the problem can have several roots. One being an issue between Python and C++ while sending actions, which is something we also encountered once, see this issue report. If nothing of the above helps, could you try port the request_data_mono script to C++?