YarpCameraBridge set_drivers_list python bindings breaks if the camera driver gets out of scope

[giotherobot]

While restructuring a python script using blf python bindings with @carloscp3009 I tried to encompass the creation of the camera bridge in a function:

def initialize_camera(param_handler):
    camera_driver = blf.robot_interface.construct_RGBD_sensor_client(
        param_handler.get_group("CAMERA_DRIVER")
    )
    camera_bridge = blf.robot_interface.YarpCameraBridge()
    camera_bridge.initialize(
        param_handler.get_group("CAMERA_DRIVER").get_group("CAMERA_BRIDGE")
    )
    if not camera_bridge.set_drivers_list([camera_driver]):
        print("Error while setting the camera driver")
        return
    return camera_bridge

This seems to not work: the camera bridge is initialized correctly but looking at the logs the ports gets deleted right after the function is executed and when trying to get the images from the camera we have a SegFault.

Returning also the camera_driver from the funciton, even without ever using it, fixes the problem ( the ports stay open).

[giotherobot]

Looking (quickly) at https://pybind11-jagerman.readthedocs.io/en/stable/advanced.html#return-value-policies we might need to change the return policy of construct_RGBD_sensor_client, but this is just a guess.

[giotherobot]

Got a minimal reproducible example going:

test_camera_driver.zip

The script is as follows:

import bipedal_locomotion_framework.bindings as blf
import yarp

def initialize_camera(param_handler):
    camera_driver = blf.robot_interface.construct_RGBD_sensor_client(
        param_handler.get_group("CAMERA_DRIVER")
    )
    camera_bridge = blf.robot_interface.YarpCameraBridge()
    camera_bridge.initialize(param_handler.get_group("CAMERA_BRIDGE"))
    if not camera_bridge.set_drivers_list([camera_driver]):
        print("Error while setting the camera driver")
        return
    return camera_bridge

if __name__ == "__main__":
    yarp.Network.init()

    param_handler = blf.parameters_handler.YarpParametersHandler()
    param_handler.set_from_filename("config/camera_config.ini")

    print(f"Parameters: {param_handler}")

    camera_bridge = initialize_camera(param_handler)

    print("Camera bridge initialized")

    # Get camera
    ret, img_rgb = camera_bridge.get_color_image("realsense")
    if not ret:
        print("RGB Camera Frame not available")
    ret, img_depth = camera_bridge.get_depth_image("realsense")
    if not ret:
        print("DEPTH Camera Frame not available")

    yarp.Network.fini()

The configuration files are: camera_config.ini:

name                    camera

[CAMERA_DRIVER]
name                     realsense
local_prefix             behaviour-generation
local_image_port_postfix /rgbImage:i
local_depth_port_postfix /depthImage:i
local_rpc_port_postfix   /rpc:o
remote_image_port        /depthCamera/rgbImage:o 
remote_depth_port        /depthCamera/depthImage:o
remote_rpc_port          /depthCamera/rpc:i
image_carrier            mjpeg
depth_carrier            fast_tcp+send.portmonitor+file.depthimage_compression_zlib+recv.portmonitor+file.depthimage_compression_zlib+type.dll

[include CAMERA_BRIDGE cameraBridgeConfig.ini]

cameraBridgeConfig.ini

stream_cameras true

[Cameras]
rgbd_cameras_list ("realsense")

And the output is:

python test_camera_driver_fcn.py 
[INFO] |yarp.os.Port|/giovanni-xps159520/python/400999/clock:i| Port /giovanni-xps159520/python/400999/clock:i active at tcp://10.240.78.160:10033/
[INFO] |yarp.os.Network| Success: port-to-port persistent connection added.
[INFO] |yarp.os.Time| Waiting for clock server to start broadcasting data ...
[INFO] |yarp.os.impl.PortCoreInputUnit|/giovanni-xps159520/python/400999/clock:i| Receiving input from /clock to /giovanni-xps159520/python/400999/clock:i using tcp
Parameters: (name camera) (CAMERA_DRIVER (name realsense) (local_prefix behaviour-generation) (local_image_port_postfix "/rgbImage:i") (local_depth_port_postfix "/depthImage:i") (local_rpc_port_postfix "/rpc:o") (remote_image_port "/depthCamera/rgbImage:o") (remote_depth_port "/depthCamera/depthImage:o") (remote_rpc_port "/depthCamera/rpc:i") (image_carrier mjpeg) (depth_carrier "fast_tcp+send.portmonitor+file.depthimage_compression_zlib+recv.portmonitor+file.depthimage_compression_zlib+type.dll")) (CAMERA_BRIDGE (stream_cameras true) (Cameras (rgbd_cameras_list (realsense))))
[DEBUG] |yarp.dev.PolyDriver|RGBDSensorClient| Parameters are (DepthCarrier "fast_tcp+send.portmonitor+file.depthimage_compression_zlib+recv.portmonitor+file.depthimage_compression_zlib+type.dll") (ImageCarrier mjpeg) (device RGBDSensorClient) (localDepthPort "/behaviour-generation/depthImage:i") (localImagePort "/behaviour-generation/rgbImage:i") (localRpcPort "/behaviour-generation/rpc:o") (remoteDepthPort "/depthCamera/depthImage:o") (remoteImagePort "/depthCamera/rgbImage:o") (remoteRpcPort "/depthCamera/rpc:i")
[INFO] |yarp.os.Port|/behaviour-generation/rgbImage:i| Port /behaviour-generation/rgbImage:i active at tcp://10.240.78.160:10006/
[INFO] |yarp.os.Port|/behaviour-generation/depthImage:i| Port /behaviour-generation/depthImage:i active at tcp://10.240.78.160:10007/
[INFO] |yarp.os.impl.PortCoreInputUnit|/behaviour-generation/rgbImage:i| Receiving input from /depthCamera/rgbImage:o to /behaviour-generation/rgbImage:i using mjpeg
[INFO] |yarp.os.impl.PortCoreInputUnit|/behaviour-generation/depthImage:i| Receiving input from /depthCamera/depthImage:o to /behaviour-generation/depthImage:i using fast_tcp+send.portmonitor+file.depthimage_compression_zlib+recv.portmonitor+file.depthimage_compression_zlib+type.dll
[INFO] |yarp.os.Port|/behaviour-generation/rpc:o| Port /behaviour-generation/rpc:o active at tcp://10.240.78.160:10008/
[INFO] |yarp.os.impl.PortCoreOutputUnit|/behaviour-generation/rpc:o| Sending output from /behaviour-generation/rpc:o to /depthCamera/rpc:i using tcp
[INFO] |yarp.dev.PolyDriver|RGBDSensorClient| Created device <RGBDSensorClient>. See C++ class RGBDSensorClient for documentation.
[DEBUG] [2024-08-07 11:09:13.738] [thread: 400999] [blf] [BipedalLocomotion::YarpUtilities::getVectorFromSearchable] Missing field named: rgb_cameras_list.
[DEBUG] [2024-08-07 11:09:13.738] [thread: 400999] [blf] [BipedalLocomotion::YarpUtilities::getVectorFromSearchable] Missing field named: rgbd_image_width.
[INFO] [2024-08-07 11:09:13.738] [thread: 400999] [blf] [YarpCameraBridge::Impl::configureCameras]  'rgb_image_width' and / or 'rgb_image_height' are not provided. The image size will be retrieved from the YARP interface.
[INFO] |yarp.os.impl.PortCoreInputUnit|/behaviour-generation/rgbImage:i| Removing input from /depthCamera/rgbImage:o to /behaviour-generation/rgbImage:i
[INFO] |yarp.os.impl.PortCoreInputUnit|/behaviour-generation/depthImage:i| Removing input from /depthCamera/depthImage:o to /behaviour-generation/depthImage:i
[INFO] |yarp.os.impl.PortCoreOutputUnit|/behaviour-generation/rpc:o| Removing output from /behaviour-generation/rpc:o to /depthCamera/rpc:i
Camera bridge initialized
pure virtual method called
terminate called without an active exception
Aborted (core dumped)

[S-Dafarra]

cc @GiulioRomualdi

Looking (quickly) at pybind11-jagerman.readthedocs.io/en/stable/advanced.html#return-value-policies we might need to change the return policy of construct_RGBD_sensor_client, but this is just a guess.

@giotherobot you could try and check if that fixes the issue