Problem with 2 D415 cameras | RS2_USB_STATUS_BUSY | Not detected after realsense-ros in other tools

I just made the mistake of updating software (dumb) and am now having trouble with a setup running two D415 cameras. My launch file had been working well for sometime. Rather than just downgrading, I would like to understand the source of the problem and fix it.

The system seems to think that the cameras are busy on startup, I do not know why. If I am running in rosmon, I kill the nodes, and then restart them, then it seems to work, although CPU usage goes crazy. Simple relaunching does not solve the problem.

Weirdly, when I turn on the computer, if I launch realsense viewer, there are no problems. If I run the ros launch file above and then open realsense viewer, the cameras do not show up.

If downgrading software is the way to go. How do I downgrade librealsense, the ros stack, and the firmware all at once? Or do I only need to downgrade one? Where should I be downgrading to?

console output:

nuc-admin@flo-nuc:~$ roslaunch flo_telepresence realsense.launch 
... logging to /home/nuc-admin/.ros/log/a622f598-8fb9-11ea-98e5-e470b83e61ec/roslaunch-flo-nuc-15444.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
WARNING: disk usage in log directory [/home/nuc-admin/.ros/log] is over 1GB.
It's recommended that you use the 'rosclean' command.

started roslaunch server http://10.153.99.5:39277/

SUMMARY
========

PARAMETERS
 * /lower_realsense/realsense2_camera/accel_fps: 250
 * /lower_realsense/realsense2_camera/accel_frame_id: lower_realsense_a...
 * /lower_realsense/realsense2_camera/accel_optical_frame_id: lower_realsense_a...
 * /lower_realsense/realsense2_camera/align_depth: False
 * /lower_realsense/realsense2_camera/aligned_depth_to_color_frame_id: lower_realsense_a...
 * /lower_realsense/realsense2_camera/aligned_depth_to_fisheye1_frame_id: lower_realsense_a...
 * /lower_realsense/realsense2_camera/aligned_depth_to_fisheye2_frame_id: lower_realsense_a...
 * /lower_realsense/realsense2_camera/aligned_depth_to_fisheye_frame_id: lower_realsense_a...
 * /lower_realsense/realsense2_camera/aligned_depth_to_infra1_frame_id: lower_realsense_a...
 * /lower_realsense/realsense2_camera/aligned_depth_to_infra2_frame_id: lower_realsense_a...
 * /lower_realsense/realsense2_camera/allow_no_texture_points: False
 * /lower_realsense/realsense2_camera/base_frame_id: lower_realsense_link
 * /lower_realsense/realsense2_camera/calib_odom_file: 
 * /lower_realsense/realsense2_camera/clip_distance: -2.0
 * /lower_realsense/realsense2_camera/color_fps: 30
 * /lower_realsense/realsense2_camera/color_frame_id: lower_realsense_c...
 * /lower_realsense/realsense2_camera/color_height: 480
 * /lower_realsense/realsense2_camera/color_optical_frame_id: lower_realsense_c...
 * /lower_realsense/realsense2_camera/color_width: 848
 * /lower_realsense/realsense2_camera/depth_fps: 30
 * /lower_realsense/realsense2_camera/depth_frame_id: lower_realsense_d...
 * /lower_realsense/realsense2_camera/depth_height: 720
 * /lower_realsense/realsense2_camera/depth_optical_frame_id: lower_realsense_d...
 * /lower_realsense/realsense2_camera/depth_width: 1280
 * /lower_realsense/realsense2_camera/enable_accel: False
 * /lower_realsense/realsense2_camera/enable_color: True
 * /lower_realsense/realsense2_camera/enable_depth: False
 * /lower_realsense/realsense2_camera/enable_fisheye1: True
 * /lower_realsense/realsense2_camera/enable_fisheye2: True
 * /lower_realsense/realsense2_camera/enable_fisheye: True
 * /lower_realsense/realsense2_camera/enable_gyro: False
 * /lower_realsense/realsense2_camera/enable_infra1: False
 * /lower_realsense/realsense2_camera/enable_infra2: False
 * /lower_realsense/realsense2_camera/enable_pointcloud: False
 * /lower_realsense/realsense2_camera/enable_sync: False
 * /lower_realsense/realsense2_camera/filters: 
 * /lower_realsense/realsense2_camera/fisheye1_frame_id: lower_realsense_f...
 * /lower_realsense/realsense2_camera/fisheye1_optical_frame_id: lower_realsense_f...
 * /lower_realsense/realsense2_camera/fisheye2_frame_id: lower_realsense_f...
 * /lower_realsense/realsense2_camera/fisheye2_optical_frame_id: lower_realsense_f...
 * /lower_realsense/realsense2_camera/fisheye_fps: 30
 * /lower_realsense/realsense2_camera/fisheye_frame_id: lower_realsense_f...
 * /lower_realsense/realsense2_camera/fisheye_height: 480
 * /lower_realsense/realsense2_camera/fisheye_optical_frame_id: lower_realsense_f...
 * /lower_realsense/realsense2_camera/fisheye_width: 640
 * /lower_realsense/realsense2_camera/gyro_fps: 400
 * /lower_realsense/realsense2_camera/gyro_frame_id: lower_realsense_g...
 * /lower_realsense/realsense2_camera/gyro_optical_frame_id: lower_realsense_g...
 * /lower_realsense/realsense2_camera/imu_optical_frame_id: lower_realsense_i...
 * /lower_realsense/realsense2_camera/infra1_frame_id: lower_realsense_i...
 * /lower_realsense/realsense2_camera/infra1_optical_frame_id: lower_realsense_i...
 * /lower_realsense/realsense2_camera/infra2_frame_id: lower_realsense_i...
 * /lower_realsense/realsense2_camera/infra2_optical_frame_id: lower_realsense_i...
 * /lower_realsense/realsense2_camera/infra_fps: 30
 * /lower_realsense/realsense2_camera/infra_height: 720
 * /lower_realsense/realsense2_camera/infra_width: 1280
 * /lower_realsense/realsense2_camera/initial_reset: False
 * /lower_realsense/realsense2_camera/json_file_path: 
 * /lower_realsense/realsense2_camera/linear_accel_cov: 0.01
 * /lower_realsense/realsense2_camera/odom_frame_id: lower_realsense_o...
 * /lower_realsense/realsense2_camera/pointcloud_texture_index: 0
 * /lower_realsense/realsense2_camera/pointcloud_texture_stream: RS2_STREAM_COLOR
 * /lower_realsense/realsense2_camera/pose_frame_id: lower_realsense_p...
 * /lower_realsense/realsense2_camera/pose_optical_frame_id: lower_realsense_p...
 * /lower_realsense/realsense2_camera/publish_odom_tf: True
 * /lower_realsense/realsense2_camera/rosbag_filename: 
 * /lower_realsense/realsense2_camera/serial_no: 904412060717
 * /lower_realsense/realsense2_camera/topic_odom_in: odom_in
 * /lower_realsense/realsense2_camera/unite_imu_method: 
 * /rosdistro: kinetic
 * /rosversion: 1.12.14
 * /upper_realsense/realsense2_camera/accel_fps: 250
 * /upper_realsense/realsense2_camera/accel_frame_id: upper_realsense_a...
 * /upper_realsense/realsense2_camera/accel_optical_frame_id: upper_realsense_a...
 * /upper_realsense/realsense2_camera/align_depth: False
 * /upper_realsense/realsense2_camera/aligned_depth_to_color_frame_id: upper_realsense_a...
 * /upper_realsense/realsense2_camera/aligned_depth_to_fisheye1_frame_id: upper_realsense_a...
 * /upper_realsense/realsense2_camera/aligned_depth_to_fisheye2_frame_id: upper_realsense_a...
 * /upper_realsense/realsense2_camera/aligned_depth_to_fisheye_frame_id: upper_realsense_a...
 * /upper_realsense/realsense2_camera/aligned_depth_to_infra1_frame_id: upper_realsense_a...
 * /upper_realsense/realsense2_camera/aligned_depth_to_infra2_frame_id: upper_realsense_a...
 * /upper_realsense/realsense2_camera/allow_no_texture_points: False
 * /upper_realsense/realsense2_camera/base_frame_id: upper_realsense_link
 * /upper_realsense/realsense2_camera/calib_odom_file: 
 * /upper_realsense/realsense2_camera/clip_distance: -2.0
 * /upper_realsense/realsense2_camera/color_fps: 30
 * /upper_realsense/realsense2_camera/color_frame_id: upper_realsense_c...
 * /upper_realsense/realsense2_camera/color_height: 480
 * /upper_realsense/realsense2_camera/color_optical_frame_id: upper_realsense_c...
 * /upper_realsense/realsense2_camera/color_width: 848
 * /upper_realsense/realsense2_camera/depth_fps: 30
 * /upper_realsense/realsense2_camera/depth_frame_id: upper_realsense_d...
 * /upper_realsense/realsense2_camera/depth_height: 720
 * /upper_realsense/realsense2_camera/depth_optical_frame_id: upper_realsense_d...
 * /upper_realsense/realsense2_camera/depth_width: 1280
 * /upper_realsense/realsense2_camera/enable_accel: False
 * /upper_realsense/realsense2_camera/enable_color: True
 * /upper_realsense/realsense2_camera/enable_depth: False
 * /upper_realsense/realsense2_camera/enable_fisheye1: True
 * /upper_realsense/realsense2_camera/enable_fisheye2: True
 * /upper_realsense/realsense2_camera/enable_fisheye: True
 * /upper_realsense/realsense2_camera/enable_gyro: False
 * /upper_realsense/realsense2_camera/enable_infra1: False
 * /upper_realsense/realsense2_camera/enable_infra2: False
 * /upper_realsense/realsense2_camera/enable_pointcloud: False
 * /upper_realsense/realsense2_camera/enable_sync: False
 * /upper_realsense/realsense2_camera/filters: 
 * /upper_realsense/realsense2_camera/fisheye1_frame_id: upper_realsense_f...
 * /upper_realsense/realsense2_camera/fisheye1_optical_frame_id: upper_realsense_f...
 * /upper_realsense/realsense2_camera/fisheye2_frame_id: upper_realsense_f...
 * /upper_realsense/realsense2_camera/fisheye2_optical_frame_id: upper_realsense_f...
 * /upper_realsense/realsense2_camera/fisheye_fps: 30
 * /upper_realsense/realsense2_camera/fisheye_frame_id: upper_realsense_f...
 * /upper_realsense/realsense2_camera/fisheye_height: 480
 * /upper_realsense/realsense2_camera/fisheye_optical_frame_id: upper_realsense_f...
 * /upper_realsense/realsense2_camera/fisheye_width: 640
 * /upper_realsense/realsense2_camera/gyro_fps: 400
 * /upper_realsense/realsense2_camera/gyro_frame_id: upper_realsense_g...
 * /upper_realsense/realsense2_camera/gyro_optical_frame_id: upper_realsense_g...
 * /upper_realsense/realsense2_camera/imu_optical_frame_id: upper_realsense_i...
 * /upper_realsense/realsense2_camera/infra1_frame_id: upper_realsense_i...
 * /upper_realsense/realsense2_camera/infra1_optical_frame_id: upper_realsense_i...
 * /upper_realsense/realsense2_camera/infra2_frame_id: upper_realsense_i...
 * /upper_realsense/realsense2_camera/infra2_optical_frame_id: upper_realsense_i...
 * /upper_realsense/realsense2_camera/infra_fps: 30
 * /upper_realsense/realsense2_camera/infra_height: 480
 * /upper_realsense/realsense2_camera/infra_width: 640
 * /upper_realsense/realsense2_camera/initial_reset: False
 * /upper_realsense/realsense2_camera/json_file_path: 
 * /upper_realsense/realsense2_camera/linear_accel_cov: 0.01
 * /upper_realsense/realsense2_camera/odom_frame_id: upper_realsense_o...
 * /upper_realsense/realsense2_camera/pointcloud_texture_index: 0
 * /upper_realsense/realsense2_camera/pointcloud_texture_stream: RS2_STREAM_COLOR
 * /upper_realsense/realsense2_camera/pose_frame_id: upper_realsense_p...
 * /upper_realsense/realsense2_camera/pose_optical_frame_id: upper_realsense_p...
 * /upper_realsense/realsense2_camera/publish_odom_tf: True
 * /upper_realsense/realsense2_camera/rosbag_filename: 
 * /upper_realsense/realsense2_camera/serial_no: 912322061173
 * /upper_realsense/realsense2_camera/topic_odom_in: odom_in
 * /upper_realsense/realsense2_camera/unite_imu_method: 

NODES
  /lower_realsense/
    realsense2_camera (nodelet/nodelet)
    realsense2_camera_manager (nodelet/nodelet)
  /upper_realsense/
    realsense2_camera (nodelet/nodelet)
    realsense2_camera_manager (nodelet/nodelet)

ROS_MASTER_URI=http://localhost:11311

process[lower_realsense/realsense2_camera_manager-1]: started with pid [15466]
process[lower_realsense/realsense2_camera-2]: started with pid [15467]
process[upper_realsense/realsense2_camera_manager-3]: started with pid [15468]
process[upper_realsense/realsense2_camera-4]: started with pid [15469]
[ INFO] [1588788685.468106682]: Initializing nodelet with 4 worker threads.
[ INFO] [1588788685.482146770]: Initializing nodelet with 4 worker threads.
[ INFO] [1588788685.520152048]: RealSense ROS v2.2.8
[ INFO] [1588788685.520177598]: Running with LibRealSense v2.33.1
[ INFO] [1588788685.533659376]: RealSense ROS v2.2.8
[ INFO] [1588788685.533695519]: Running with LibRealSense v2.33.1
 06/05 14:11:25,661 ERROR [139714514368256] (handle-libusb.h:95) failed to claim usb interface: 0, error: RS2_USB_STATUS_BUSY
 06/05 14:11:25,662 WARNING [139756541302528] (uvc-device.cpp:664) interrupt event received: 1, 3, 0, 7, 0, 13, 
 06/05 14:11:25,666 WARNING [139756541302528] (uvc-device.cpp:664) interrupt event received: 1, 3, 0, 7, 0, 20, 
 06/05 14:11:25,682 WARNING [139714547939072] (rs.cpp:283) null pointer passed for argument "device"
terminate called after throwing an instance of 'rs2::error'
  what():  failed to set power state
[lower_realsense/realsense2_camera_manager-1] process has died [pid 15466, exit code -6, cmd /opt/ros/kinetic/lib/nodelet/nodelet manager __name:=realsense2_camera_manager __log:=/home/nuc-admin/.ros/log/a622f598-8fb9-11ea-98e5-e470b83e61ec/lower_realsense-realsense2_camera_manager-1.log].
log file: /home/nuc-admin/.ros/log/a622f598-8fb9-11ea-98e5-e470b83e61ec/lower_realsense-realsense2_camera_manager-1*.log
[ INFO] [1588788685.919835448]: getParameters...
[ INFO] [1588788685.986562489]: setupDevice...
[ INFO] [1588788685.986592806]: JSON file is not provided
[ INFO] [1588788685.986611824]: ROS Node Namespace: upper_realsense
[ INFO] [1588788685.986633518]: Device Name: Intel RealSense D415
[ INFO] [1588788685.986645240]: Device Serial No: 912322061173
[ INFO] [1588788685.986656444]: Device FW version: 05.12.03.00
[ INFO] [1588788685.986667604]: Device Product ID: 0x0AD3
[ INFO] [1588788685.986680093]: Enable PointCloud: Off
[ INFO] [1588788685.986692181]: Align Depth: Off
[ INFO] [1588788685.986705761]: Sync Mode: Off
[ INFO] [1588788685.986744254]: Device Sensors: 
[ INFO] [1588788685.986779882]: Stereo Module was found.
[ INFO] [1588788685.986798221]: RGB Camera was found.
[ INFO] [1588788685.986825867]: (Fisheye, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1588788685.986837817]: (Fisheye, 1) sensor isn't supported by current device! -- Skipping...
[ INFO] [1588788685.986851616]: (Fisheye, 2) sensor isn't supported by current device! -- Skipping...
[ INFO] [1588788685.986863020]: (Pose, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1588788685.986885567]: num_filters: 0
[ INFO] [1588788685.986897894]: Setting Dynamic reconfig parameters.
[ INFO] [1588788691.143280503]: Done Setting Dynamic reconfig parameters.
[ INFO] [1588788691.171650707]: color stream is enabled - width: 848, height: 480, fps: 30
[ INFO] [1588788691.171804990]: setupPublishers...
[ INFO] [1588788691.175239860]: Expected frequency for color = 30.00000
[ INFO] [1588788691.209106475]: setupStreams...
[ INFO] [1588788691.260165542]: insert Color to RGB Camera
[ INFO] [1588788691.326572559]: SELECTED BASE:Depth, 0
[ INFO] [1588788691.368299520]: RealSense Node Is Up!
 06/05 14:11:31,800 WARNING [139756516124416] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 06/05 14:11:31,861 WARNING [139756516124416] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
[ WARN] [1588788692.198603124]: Hardware Notification:RT IC2 Config error,1.58879e+12,Error,Hardware Error
[ERROR] [1588788692.198635568]: Hardware Reset is needed. use option: initial_reset:=true
 06/05 14:11:32,725 WARNING [139756516124416] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 06/05 14:11:32,789 WARNING [139756516124416] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 06/05 14:11:33,154 WARNING [139756516124416] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 06/05 14:11:33,221 WARNING [139756516124416] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 06/05 14:11:33,224 WARNING [139756516124416] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 06/05 14:11:33,285 WARNING [139756516124416] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 06/05 14:11:33,289 WARNING [139756516124416] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 06/05 14:11:33,313 ERROR [139756021217024] (uvc-streamer.cpp:106) uvc streamer watchdog triggered on endpoint: 132
 06/05 14:11:33,314 WARNING [139756516124416] (messenger-libusb.cpp:92) usb_request_queue returned error, endpoint: 135 error: No such device, number: 19
 06/05 14:11:33,318 WARNING [139756021217024] (messenger-libusb.cpp:30) reset_endpoint returned error, index: 132, error: No such device, number: 19
 06/05 14:11:33,318 WARNING [139756021217024] (messenger-libusb.cpp:92) usb_request_queue returned error, endpoint: 132 error: No such device, number: 19
 06/05 14:11:33,318 ERROR [139756021217024] (uvc-streamer.cpp:138) failed to submit UVC request, error: -13
 06/05 14:11:33,318 WARNING [139756021217024] (messenger-libusb.cpp:92) usb_request_queue returned error, endpoint: 132 error: No such device, number: 19
 06/05 14:11:33,318 ERROR [139756021217024] (uvc-streamer.cpp:138) failed to submit UVC request, error: -13
 06/05 14:11:33,345 WARNING [139756516124416] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No such device, number: 19
 06/05 14:11:33,349 WARNING [139756516124416] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No such device, number: 19
 06/05 14:11:33,356 ERROR [139756532909824] (global_timestamp_reader.cpp:188) Error during time_diff_keeper polling: usb device disconnected
 06/05 14:11:33,359 WARNING [139756516124416] (messenger-libusb.cpp:105) usb_request_cancel returned error, endpoint: 135 error: No such device, number: 19
 06/05 14:11:33,369 ERROR [139756415477504] (error-handling.cpp:68) Error during polling error handler: usb device disconnected
 06/05 14:11:33,456 ERROR [139756516124416] (handle-libusb.h:51) failed to open usb interface: 0, error: RS2_USB_STATUS_NO_DEVICE
 06/05 14:11:33,466 ERROR [139756532909824] (global_timestamp_reader.cpp:188) Error during time_diff_keeper polling: failed to set power state
[lower_realsense/realsense2_camera-2] process has finished cleanly
log file: /home/nuc-admin/.ros/log/a622f598-8fb9-11ea-98e5-e470b83e61ec/lower_realsense-realsense2_camera-2*.log
[ERROR] [1588788695.919472157]: The device has been disconnected!
 06/05 14:11:39,582 WARNING [139756532909824] (types.cpp:49) set_xu(ctrl=1) failed! Last Error: Success
 06/05 14:11:39,582 ERROR [139756532909824] (global_timestamp_reader.cpp:188) Error during time_diff_keeper polling: set_xu(ctrl=1) failed! Last Error: Success
 06/05 14:11:39,702 WARNING [139756532909824] (types.cpp:49) hwmon command 0x1 failed. Error type: Success (0).
 06/05 14:11:39,702 ERROR [139756532909824] (global_timestamp_reader.cpp:188) Error during time_diff_keeper polling: hwmon command 0x1 failed. Error type: Success (0).
 06/05 14:11:40,385 WARNING [139756415477504] (types.cpp:49) get_xu(id=7) failed! Last Error: Success
 06/05 14:11:40,385 ERROR [139756415477504] (error-handling.cpp:68) Error during polling error handler: get_xu(id=7) failed! Last Error: Success
 06/05 14:11:45,817 WARNING [139756532909824] (types.cpp:49) set_xu(ctrl=1) failed! Last Error: Success
 06/05 14:11:45,817 ERROR [139756532909824] (global_timestamp_reader.cpp:188) Error during time_diff_keeper polling: set_xu(ctrl=1) failed! Last Error: Success
 06/05 14:11:45,831 WARNING [139756021217024] (messenger-libusb.cpp:105) usb_request_cancel returned error, endpoint: 132 error: No such device, number: 19
 06/05 14:11:45,831 WARNING [139756021217024] (messenger-libusb.cpp:105) usb_request_cancel returned error, endpoint: 132 error: No such device, number: 19
 06/05 14:11:45,831 WARNING [139756021217024] (messenger-libusb.cpp:30) reset_endpoint returned error, index: 132, error: No such device, number: 19
[ INFO] [1588788705.864227772]: Checking new devices...
 06/05 14:11:46,014 WARNING [139756390299392] (uvc-device.cpp:664) interrupt event received: 1, 3, 0, 7, 0, 0, 
 06/05 14:11:46,018 WARNING [139756390299392] (uvc-device.cpp:664) interrupt event received: 1, 3, 0, 7, 0, 0, 
[ INFO] [1588788706.273219629]: getParameters...
[ INFO] [1588788706.342231199]: setupDevice...
[ INFO] [1588788706.342261910]: JSON file is not provided
[ INFO] [1588788706.342277250]: ROS Node Namespace: upper_realsense
[ INFO] [1588788706.342293755]: Device Name: Intel RealSense D415
[ INFO] [1588788706.342307046]: Device Serial No: 912322061173
[ INFO] [1588788706.342320897]: Device FW version: 05.12.03.00
[ INFO] [1588788706.342333690]: Device Product ID: 0x0AD3
[ INFO] [1588788706.342346160]: Enable PointCloud: Off
[ INFO] [1588788706.342358541]: Align Depth: Off
[ INFO] [1588788706.342370751]: Sync Mode: Off
[ INFO] [1588788706.342388953]: Device Sensors: 
[ INFO] [1588788706.342405723]: Stereo Module was found.
[ INFO] [1588788706.342444569]: RGB Camera was found.
[ INFO] [1588788706.342463582]: (Fisheye, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1588788706.342477531]: (Fisheye, 1) sensor isn't supported by current device! -- Skipping...
[ INFO] [1588788706.342490622]: (Fisheye, 2) sensor isn't supported by current device! -- Skipping...
[ INFO] [1588788706.342503650]: (Pose, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1588788706.342519065]: num_filters: 0
[ INFO] [1588788706.342531574]: Setting Dynamic reconfig parameters.
[ INFO] [1588788711.496683351]: Done Setting Dynamic reconfig parameters.
[ INFO] [1588788711.539592516]: color stream is enabled - width: 848, height: 480, fps: 30
[ INFO] [1588788711.540301334]: setupPublishers...
[ INFO] [1588788711.543133283]: Expected frequency for color = 30.00000
[ INFO] [1588788711.574979698]: setupStreams...
[ INFO] [1588788711.625969888]: insert Color to RGB Camera
[ INFO] [1588788711.692674013]: SELECTED BASE:Depth, 0
[ INFO] [1588788711.738266856]: RealSense Node Is Up!
 06/05 14:11:52,166 WARNING [139756524517120] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 06/05 14:11:52,227 WARNING [139756524517120] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
^C[upper_realsense/realsense2_camera-4] killing on exit
[upper_realsense/realsense2_camera_manager-3] killing on exit
 06/05 14:33:16,176 WARNING [139757266376576] (types.cpp:49) stop_streaming() failed. UVC device is not streaming!
terminate called after throwing an instance of 'rs2::wrong_api_call_sequence_error'
  what():  stop_streaming() failed. UVC device is not streaming!
shutting down processing monitor...
... shutting down processing monitor complete
done
nuc-admin@flo-nuc:~$ alsamixer^C

I upgraded the version of realsense ros that I am using and downgraded the version of librealsense. Combined with unplugging and replugging in cables (gotta try everything), it now seems to be working as expected again.

I would love to see some guidance from intel on how to deploy their code safely. Running an apt update/upgrade should never break things. But with realsense, it happens all the time. On whole the platform seems super unstable. Is that a design problem, a user problem, both?

It is recommended that the RealSense ROS wrapper is matched with a particular version of the Librealsense SDK where possible. On the 'Releases' page for the wrapper, each version lists a recommended Librealsense SDK version to use it with.

https://github.com/IntelRealSense/realsense-ros/releases

The log in your first message says that you were using ROS wrapper 2.2.8 with Librealsense SDK 2.33.1. According to the Releases page, the appropriate match for wrapper version 2.2.8 would be the older Librealsense SDK version 2.25.0.

It is sometimes possible to 'mix and match' non-recommended combinations of wrapper and SDK, but in general the best way to make a good match between wrapper and SDK is to follow the recommended SDK for a particular wrapper version.

In regard to stability, the RealSense SDK aims to run on anything and is extremely adaptable in regard to the hardware and software that it can be used with. There are sometimes hardware and / or software configurations that require a bit more work to achieve a stable RealSense setup on though. These cases are usually mostly resolved over time as the RealSense community, the RealSense team or both develop guides and code solutions to overcome problems.

It is also worth bearing in mind that the ROS wrapper and Librealsense are not developed in sync, so there may be a delay between the release of a new Librealsense version and a ROS wrapper version that officially supports it (though the ROS wrapper for the previous Librealsense version may still work with the newer Librealsense release).

@MartyG-RealSense I agree, that mismatch was likely the source of the problem. Updating a system by looking at a webpage isn't a great solution. It is pretty easy to programatically update the realsense-ros repo and do the rebuilds, etc. Is there a way to programmatically install the recommended version of librealsense? Ideally there would just be an install-realsense-ros script and an update-realsense-ros script.

The set of installation instructions in the link below may provide useful insights.

https://github.com/IntelRealSense/realsense-ros/blob/development/.travis.yml

The instructions in this file are summarised by a two-step description in the link below, in which Librealsense is installed first and the ROS wrapper second:

https://github.com/IntelRealSense/realsense-ros#method-2-the-realsense-distribution

This could create a "chicken and the egg" dilemma ... if the Librealsense SDK is installed first and the ROS wrapper second, then the ROS wrapper version that was auto-selected to be installed would have to be matched to the Librealsense version installed in the previous step in order to get a recommended compatibility match. This may be why manually checking the Releases page is the most practical solution in most cases, if not the ideal one.

Yeah, I don't see how those scripts prevent one from getting a version of librealsense that is ahead of the ROS wrapper. Regardless my issue is fixed, so I am going to close this issue.

A feature request before I go though: If improving the installation experience is out of scope, then adding a check to the ros wrapper's startup that ensures that the librealsense version being used is the ideal one should be added. That check should not break anything but should make a very loud (but silenceable through parameters) noise to let the user know.

Could you post your feature request as a new question on its own please? Then I can label it as such and it can be left open to be tracked by the ROS wrapper developers. Thanks!

Thanks very much! I will add the label to it now.

@MartyG-RealSense, having this problem again, but I don't think it is due to a version mismatch:

Any thoughts?

I'm using 2 cameras, but if I plug in only the problematic one, I get:

If I have nothing plugged in, I get:

So the camera connects to some degree, but there is this power state issue. That seems to pop up a lot in the issues. What is that and how is it solved?

Hi @mjsobrep Here are some things to try:

Add the term initial_reset:=true. to your roslaunch statement if you are not doing so already to see if causing the camera to reset upon roslaunch makes a difference
If you are using a serial number in your launch statement and are testing just one camera, leave the serial number out as it is not needed when you only have one camera plugged in
If you are using a launch file meant for multiple cameras when using both cameras - such as rs_multiple_devices.launch - then test with one camera using a roslaunch statement suited to one camera: roslaunch realsense2_camera rs_camera.launch
Try the USB cable from your working camera with the problematic one.
Try the problematic camera in a different USB port if you have one available
Remove the micro end of the USB cable from the side of the camera, turn the connector upside down and re-insert it into the side of the camera (it is a two-way fit at the micro end)
If the camera is plugged into a USB hub, try booting the computer with it plugged into one of the computer's built-in USB ports if you have one available. Even a USB 2.0 port would be fine if that is all you have vacant

Hi @MartyG-RealSense, thanks for the suggestions:

Added initial reset true, doesn't seem to fix the problem, but I'm going to leave it on
In the above images, the problem was even showing up with rs-enumerate-devices, which is not launch file dependent.
For whatever reason, after my ?th reboot, the cameras which are working switched, so I am thinking it is not a cable issue, since at different times they both have worked
my cameras plug in directly to the computer, into USB SuperSpeed-10 ports

I'm seeing a lot of this sort of spam in ROS, ...

This is the launch file for my realsense system

I am sometimes able to open both cameras in rs-viewer, but other times not.

Both cameras have been updated to the latest firmware through rs-viewer.

Actually, it is not just restarting the computer, just re running the launch file can cause a change in which camera chooses to work.

It seems like after a while, realsense-ros stops trying to connect to the camera, sets up the publishers, and just sits there quietly, not doing anything.

Apologies for the delay in responding further, as I was carefully considering your case.

Do you have more than one program running at the same time that accesses the same camera? For example, launching the RealSense ROS wrapper and then the Viewer, or launching the Viewer first and then the ROS wrapper?

If the camera is already "claimed" by a program that has an active stream when you try to access the same camera with another program then the second program may fail to access the camera.

Never at the same time. There errors persist even after a computer restart. I'm not sure if the text of the errors is the same or not whether the camera has been used previously or if coming off a restart.

Further research found a past case that had similar errors to this case. The case was about launching D435 and T265 together which is not the same situation as your one with two D415s but is still an interesting reference.

https://support.intelrealsense.com/hc/en-us/community/posts/360034992254-t265-and-d435-Failed-to-claim-USB-interface-error

One RealSense ROS user with problems launching two cameras wrote some arg code for delaying the launch of one of the cameras.

https://github.com/IntelRealSense/realsense-ros/issues/774#issuecomment-493933596

I note though that you have also had problems with sometimes recognizing the cameras in the Viewer, which is not related to the ROS launch process.

Hmm, those are interesting, I will try to see if some sort of delaying helps. Is there a large power draw at startup for the cameras?

When shutting down, I have noticed that one of the realsense nodes sometimes closes messily with a sigterm

This is an example of the not working camera thinking it is connected: It is not, looking at the color image_raw and depth image_rect images:

It appears that the cameras do sometimes work, but I don't yet have a feel for what might be causing it to sometimes work or the frequency of that.

I'm not sure how much power the camera draws during roslaunch. If streams are being started and published then one might expect it to be more than if the camera was detected in the Viewer but not currently streaming. The recommendation that I know of when calculating the camera power needs of a project is to budget for around 2W power draw per camera.

That is why using a mains electricity powered hub is useful for multiple camera projects, as they can supply around 12W of power.

I am away from computer now for 7 hours. Good luck!

After a night of leaving the cameras and computer off, things seemed to be working more consistently. I do not know why that might be. I lowered the resolution on the color imagers to 1080x720.

Just tried changing to 1920x1080 and it didn't work. Changing back to 1080x720 did again work, although after a lot of errors and a tough time finding the cameras.

So running these sequentially	trying	outcome
Reboot and run at 1920x1080 with initial reset true	one camera connects (does seem to change which camera is working)
Reboot and run at 1920x1080 with initial reset false	neither camera works, looks like nodes actually crashed...
run at 1080x720 with initial reset false, no reboot	neither camera works, getting "No Realsense devices were found!" warning
run at 1080x720 with initial reset true, no reboot	neither camera works, getting "No Realsense devices were found!" warning
Reboot and run at 1080x720 with initial reset false	neither camera works, getting "No Realsense devices were found!" warning
continuing from above, unplug and plug in both cameras	Both cameras now work
stop nodes, restart at 1080x720 with initial reset false, no reboot	Both cameras work
reboot and run at 1080x720 with initial reset false	Nodes ended up crashing, see out1 below
stop nodes, restart at 1080x720 with initial reset false, no reboot	neither camera works, getting "No Realsense devices were found!" warning
stop nodes, restart at 1080x720 with initial reset true, no reboot	neither camera works, getting "No Realsense devices were found!" warning
continuing from above, unplug and plug in both cameras	One camera works, the other crashed with out2
reboot and run at 1080x720 with initial reset true	both cameras work
reboot and run at 1080x720 with initial reset true	both cameras work
reboot and run at 1080x720 with initial reset true	one camera crashes (out3) and the other can't be found (out4)
flip the camera's ports (another issue somewhere mentions a bug requiring serial numbers to be ascending by port number), reboot, run at 1080x720 with initial reset true	both cameras work
reboot and run at 1080x720 with initial reset true	one camera works, the other crashes (out5)
try plugging into a usb hub with max 2.1A available (this limitation is why I haven't used it), reboot, run at 1080x720 with initial reset true	one camera works one doesn't
try plugging one camera into the hub one into the computer, reboot, run at 1080x720, initial reset true	both cameras work
continue, reboot, 1080x720, initial reset true	one camera works
plug cameras back into computer, 1080x720, initial reset true, start cameras after everything else, stagger start using bash sleep (so need to move to ros2 :'( )	both cameras work
repeat last	both cameras work
repeat last	both cameras work

I'm not sure how to setup the timed launch system you mention above to run launch files instead of nodes, need to figure out if that can be done. Tomorrow I will investigate whether the timing delay for starting the cameras is needed or if just starting them under different terminals is what is needed. If timing is what is important then it would be good to understand from your end why that is, do you have published power draw curves for starting up and running the cameras? Is there something in software that conflicts? Tomorrow I will also try to run at full resolution to see if this staggered start works with that.

out1:

 18/08 18:54:30,771 ERROR [140491492402944] (handle-libusb.h:95) failed to claim usb interface: 0, error: RS2_USB_STATUS_BUSY
 18/08 18:54:30,794 WARNING [140491525973760] (rs.cpp:304) null pointer passed for argument "device"
[ WARN] [1597791270.794228307]: Device 1/2 failed with exception: failed to set power state
[ WARN] [1597791270.821031122]: Asked to publish at 'fps' (30) which is higher than the 'set_camera_fps' (15), we can't publish faster than the camera provides images.

.....

18/08 18:54:50,594 WARNING [139845309544192] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 18/08 18:54:50,612 WARNING [140491484010240] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 18/08 18:54:50,612 WARNING [140491525973760] (types.cpp:78) set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
[ERROR] [1597791290.613241399]: An exception has been thrown: set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
terminate called after throwing an instance of 'rs2::invalid_value_error'
 18/08 18:54:50,613 WARNING [139845458224896] (types.cpp:78) set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
  what():  set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
[ERROR] [1597791290.614003341]: An exception has been thrown: set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
terminate called after throwing an instance of 'rs2::invalid_value_error'
  what():  set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
[lower_realsense/realsense2_camera_manager-23] process has died [pid 2964, exit code -6, cmd /opt/ros/kinetic/lib/nodelet/nodelet manager __name:=realsense2_camera_manager __log:=/home/nuc-admin/.ros/log/
c4fa8b00-e1a5-11ea-a88a-982cbccfb098/lower_realsense-realsense2_camera_manager-23.log].
log file: /home/nuc-admin/.ros/log/c4fa8b00-e1a5-11ea-a88a-982cbccfb098/lower_realsense-realsense2_camera_manager-23*.log
[upper_realsense/realsense2_camera_manager-27] process has died [pid 3001, exit code -6, cmd /opt/ros/kinetic/lib/nodelet/nodelet manager __name:=realsense2_camera_manager __log:=/home/nuc-admin/.ros/log/
c4fa8b00-e1a5-11ea-a88a-982cbccfb098/upper_realsense-realsense2_camera_manager-27.log].
log file: /home/nuc-admin/.ros/log/c4fa8b00-e1a5-11ea-a88a-982cbccfb098/upper_realsense-realsense2_camera_manager-27*.log
[upper_realsense/realsense2_camera-28] process has finished cleanly
log file: /home/nuc-admin/.ros/log/c4fa8b00-e1a5-11ea-a88a-982cbccfb098/upper_realsense-realsense2_camera-28*.log
[lower_realsense/realsense2_camera-24] process has finished cleanly
log file: /home/nuc-admin/.ros/log/c4fa8b00-e1a5-11ea-a88a-982cbccfb098/lower_realsense-realsense2_camera-24*.l

out2:

[ INFO] [1597791807.439703133]: setupDevice...
[ INFO] [1597791807.439730552]: JSON file is not provided
[ INFO] [1597791807.439745257]: ROS Node Namespace: lower_realsense
[ INFO] [1597791807.439760414]: Device Name: Intel RealSense D415
[ INFO] [1597791807.439773509]: Device Serial No: 904412060717
[ INFO] [1597791807.439788916]: Device physical port: 2-3-21
[ INFO] [1597791807.439801651]: Device FW version: 05.12.06.00
[ INFO] [1597791807.439811870]: Device Product ID: 0x0AD3
[ INFO] [1597791807.439821559]: Enable PointCloud: Off
[ INFO] [1597791807.439833162]: Align Depth: On
[ INFO] [1597791807.439844702]: Sync Mode: On
[ INFO] [1597791807.439889529]: Device Sensors: 
 18/08 19:03:27,439 ERROR [140328753407744] (handle-libusb.h:95) failed to claim usb interface: 0, error: RS2_USB_STATUS_BUSY
 18/08 19:03:27,450 WARNING [140328900060928] (rs.cpp:360) null pointer passed for argument "list"
[ERROR] [1597791807.450248494]: An exception has been thrown: failed to set power state
terminate called after throwing an instance of 'rs2::error'
  what():  failed to set power state
[lower_realsense/realsense2_camera_manager-23] process has died [pid 10824, exit code -6, cmd /opt/ros/kinetic/lib/nodelet/nodelet manager __name:=realsense2_camera_manager __log:=/home/nuc-admin/.ros/lo$
/c4fa8b00-e1a5-11ea-a88a-982cbccfb098/lower_realsense-realsense2_camera_manager-23.log].
log file: /home/nuc-admin/.ros/log/c4fa8b00-e1a5-11ea-a88a-982cbccfb098/lower_realsense-realsense2_camera_manager-23*.log
[ INFO] [1597791807.691790266]: Device with serial number 904412060717 was found.

out3:

[ INFO] [1597792570.880249894]: Device with serial number 912322061173 was found.

[ INFO] [1597792570.880279117]: Device with physical ID 2-4-4 was found.
[ INFO] [1597792570.880287386]: Device with name Intel RealSense D415 was found.
[ INFO] [1597792570.880719944]: Device with port number 2-4 was found.
[ INFO] [1597792570.882529422]: getParameters...
[ INFO] [1597792570.943920926]: setupDevice...
[ INFO] [1597792570.943954667]: JSON file is not provided
[ INFO] [1597792570.943969764]: ROS Node Namespace: upper_realsense
[ INFO] [1597792570.943985418]: Device Name: Intel RealSense D415
[ INFO] [1597792570.943993065]: Device Serial No: 912322061173
[ INFO] [1597792570.943999906]: Device physical port: 2-4-4
[ INFO] [1597792570.944006978]: Device FW version: 05.12.06.00
[ INFO] [1597792570.944013946]: Device Product ID: 0x0AD3
[ INFO] [1597792570.944020936]: Enable PointCloud: Off
[ INFO] [1597792570.944027398]: Align Depth: On
[ INFO] [1597792570.944036343]: Sync Mode: On
[ INFO] [1597792570.944117104]: Device Sensors: 
 18/08 19:16:10,944 ERROR [140436790286080] (handle-libusb.h:95) failed to claim usb interface: 0, error: RS2_USB_STATUS_BUSY
 18/08 19:16:10,954 WARNING [140436944922368] (rs.cpp:360) null pointer passed for argument "list"
[ERROR] [1597792570.954680004]: An exception has been thrown: failed to set power state
terminate called after throwing an instance of 'rs2::error'
  what():  failed to set power state
[upper_realsense/realsense2_camera_manager-27] process has died [pid 2850, exit code -6, cmd /opt/ros/kinetic/lib/nodelet/nodelet manager __name:=realsense2_camera_manager __log:=/home/nuc-admin/.ros/log$
c41f06c2-e1a8-11ea-91ac-982cbccfb098/upper_realsense-realsense2_camera_manager-27.log].
log file: /home/nuc-admin/.ros/log/c41f06c2-e1a8-11ea-91ac-982cbccfb098/upper_realsense-realsense2_camera_manager-27*.log
[ INFO] [1597792571.130816545]: Device with serial number 912322061173 was found.

[ INFO] [1597792571.130856141]: Device with physical ID 2-4-4 was found.
[ INFO] [1597792571.130887521]: Device with name Intel RealSense D415 was found.
[ INFO] [1597792571.131294457]: Device with port number 2-4 was found.
[ERROR] [1597792571.152285131]: The requested device with serial number 904412060717 is NOT found. Will Try again.
[upper_realsense/realsense2_camera-28] process has finished cleanly
log file: /home/nuc-admin/.ros/log/c41f06c2-e1a8-11ea-91ac-982cbccfb098/upper_realsense-realsense2_camera-28*.log
[ INFO] [1597792577.198455498]:  
 18/08 19:16:17,210 ERROR [140436698031872] (handle-libusb.h:95) failed to claim usb interface: 0, error: RS2_USB_STATUS_BUSY
 18/08 19:16:17,231 WARNING [140436827858688] (rs.cpp:304) null pointer passed for argument "device"

out4

This just keeps repeating:

[ INFO] [1597792727.138558932]: Device with physical ID 2-4-4 was found.
[ INFO] [1597792727.138579174]: Device with name Intel RealSense D415 was found.
[ INFO] [1597792727.139113313]: Device with port number 2-4 was found.
[ERROR] [1597792727.160540464]: The requested device with serial number 904412060717 is NOT found. Will Try again.
[ INFO] [1597792733.209254157]:  
 18/08 19:18:53,222 ERROR [140436698031872] (handle-libusb.h:95) failed to claim usb interface: 0, error: RS2_USB_STATUS_BUSY
 18/08 19:18:53,243 WARNING [140436827858688] (rs.cpp:304) null pointer passed for argument "device"
[ WARN] [1597792733.243276919]: Device 1/2 failed with exception: failed to set power state
[ INFO] [1597792733.620774728]: Device with serial number 912322061173 was found.

[ INFO] [1597792733.620800564]: Device with physical ID 2-4-4 was found.
[ INFO] [1597792733.620808152]: Device with name Intel RealSense D415 was found.
[ INFO] [1597792733.621159204]: Device with port number 2-4 was found.
[ERROR] [1597792733.642218041]: The requested device with serial number 904412060717 is NOT found. Will Try again.
[ERROR] [1597792737.586395839]: Kobuki : malformed sub-payload detected. [25][170][19 AA 55 4D 01 0F ]
[ INFO] [1597792739.690636658]:  
 18/08 19:18:59,703 ERROR [140436706424576] (handle-libusb.h:95) failed to claim usb interface: 0, error: RS2_USB_STATUS_BUSY
 18/08 19:18:59,724 WARNING [140436827858688] (rs.cpp:304) null pointer passed for argument "device"
[ WARN] [1597792739.724495557]: Device 1/2 failed with exception: failed to set power state
[ INFO] [1597792740.120868031]: Device with serial number 912322061173 was found.

[ INFO] [1597792740.120903760]: Device with physical ID 2-4-4 was found.
[ INFO] [1597792740.120914623]: Device with name Intel RealSense D415 was found.
[ INFO] [1597792740.121461940]: Device with port number 2-4 was found.
[ERROR] [1597792740.142978243]: The requested device with serial number 904412060717 is NOT found. Will Try again.
[ INFO] [1597792746.192939340]:  
 18/08 19:19:06,206 ERROR [140436689639168] (handle-libusb.h:95) failed to claim usb interface: 0, error: RS2_USB_STATUS_BUSY
 18/08 19:19:06,226 WARNING [140436827858688] (rs.cpp:304) null pointer passed for argument "device"
[ WARN] [1597792746.226949295]: Device 1/2 failed with exception: failed to set power state
[ INFO] [1597792746.618079961]: Device with serial number 912322061173 was found.

out5

 18/08 19:39:03,730 WARNING [139913949128448] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Device or resource busy, number: 16
 18/08 19:39:03,790 WARNING [139913949128448] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Device or resource busy, number: 16
 18/08 19:39:03,851 WARNING [139913982699264] (types.cpp:78) set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
 18/08 19:39:03,874 WARNING [139913982699264] (rs.cpp:304) null pointer passed for argument "device"
[ WARN] [1597793943.874385919]: Device 1/2 failed with exception: set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
[upper_realsense/realsense2_camera_manager-27] process has died [pid 2995, exit code -11, cmd /opt/ros/kinetic/lib/nodelet/nodelet manager __name:=realsense2_camera_manager __log:=/home/nuc-admin/.ros/lo$
/fa9fb2de-e1ab-11ea-ac81-982cbccfb098/upper_realsense-realsense2_camera_manager-27.log].
log file: /home/nuc-admin/.ros/log/fa9fb2de-e1ab-11ea-ac81-982cbccfb098/upper_realsense-realsense2_camera_manager-27*.log
[ INFO] [1597793944.362110477]:  
 18/08 19:39:04,374 ERROR [140144354014976] (handle-libusb.h:95) failed to claim usb interface: 0, error: RS2_USB_STATUS_BUSY
 18/08 19:39:04,395 WARNING [140144395978496] (rs.cpp:304) null pointer passed for argument "device"
[ WARN] [1597793944.395339391]: Device 1/2 failed with exception: failed to set power state
──────────────────────────────────────────────────────────────────────────────────┬─────────────

Thanks very much for the very detailed testing report!

With ROS, the multicam launch timing issue particularly affected a pairing of D435 and T265 cameras. If D435 launched first then T265 could not be recognized properly. https://github.com/IntelRealSense/realsense-ros/issues/774 that linked to earlier sums this phenomenon up well.

Would you say that you get the most frequent success when both cameras are plugged into the computer (instead of both in the hub or one in hub and one in computer)?

There was a recent case where a RealSense user had no problems when both cameras were plugged into the full size USB ports on their computer but had problems if using the micro USB-C port on the computer. They found that the ports that worked all used an Intel USB controller with xHCI support. The USB-C port that didn't work used an Asmedia USB controller without xHCI support.

https://github.com/IntelRealSense/librealsense/issues/6899#issuecomment-667720873

The workaround that they eventually found was to create a script that rebinds the USB driver.

https://github.com/IntelRealSense/librealsense/issues/6899#issuecomment-671984909

You're welcome, hopefully it can help in tracking this all down.

I read through that report. Since both of my cameras are D415s, I'm not sure how to interpret that. There are reports elsewhere in the issues about there being a sensitivity to starting the D415s in order of their serial number or plugging them in the order of their serial number.

The hub plugs into a thunderbolt port and is powered by a battery pack (the computer, an intel nuc is also powered by the same battery pack). There are a lot of other devices plugged into the hub (described in this paper) so I an concerned about stability of a camera being plugged into it. I have not noticed an improvement by plugging into the hub. We know that the realsense pushes USB pretty hard, that is why we need nice cables and the like. I have no confidence that a commodity hub off Amazon has the same quality of connection as the high end cables we all use. I would hope that whatever is breaking out USB inside of a computer (or maybe the nuc has ports with dedicated controllers?) is higher quality than an off the shelf hub. So for those reasons, I think I am going to avoid using the hub for now.

That is interesting. Wouldn't restarting the computer prevent that sort of rebinding from being needed? Once I get it all working after a reboot then perhaps I will take a look at starting after stopping.

I think there is some underlying engineering work that could be done on this to characterize the sensors and how to properly start them up. For a long period of time, I have been using an older NUC with this launch file (with the custom record param off). So I have been running with just color at a moderate resolution. I recently upgraded to a newer nuc with much better compute capabilities. Now using depth with higher resolution on the color than before.

When Intel wrote their white paper on multiple cameras for the 400 Series cameras, they tested with the AmazonBasics 4-port mains powered hub from Amazon. I bought the 7-port version of it based on that recommendation and have had no problems at all with it.

I have seen a couple of reports in the past of expensive USB 3.2 industrial grade hubs having problems with RealSense, whilst a cheap USB 3.1 hub worked fine. So industrial-grade may not be a guaranteed solution if there is a conflict with another aspect of the product (such as the USB 3.2 standard).

I believe that the issue of needing to insert cameras in a certain order mostly applies to Nvidia Jetson boards rather than NUCs.

In regard to an earlier question about power draw: the data sheet document for the 400 Series has extreme levels of detail about power delivery. If you open the PDF and do a search operation for the word power then that will help you to browse quickly through all the references.

https://dev.intelrealsense.com/docs/intel-realsense-d400-series-product-family-datasheet

Regarding the rebinding: my knowledge is hazy on that aspect, but I believe that there can be sections in the USB section of the computer's BIOS boot-up settings that specify whether a port has xHCI support. BIOSes tend to vary widely between different computers though depending on what chipset the computer is using.

I have a NUC myself as my main personal workstation, a NUC8i5INH with Radeon graphics. It has been excellent for me.

https://www.intel.co.uk/content/www/uk/en/products/boards-kits/nuc/mini-pcs/nuc8i5inh.html

Hmm, so do you know if the realsense cameras have more trouble with USB 3.2 ports compared to USB 3.1? The new nuc that I am using is all USB 3.2 ports.

Looking through the spec sheet. I see a number of places where power sequencing is discussed, as well as steady state power consumption. No discussion of any sort of higher turn on current, maybe because it doesn't exist. It would be interesting to see some real world plots of power consumption during startup of the cameras, but I am more and more thinking that is not the problem.

Do you know what the ideal settings are for the USB controllers in the BIOS (of course every bios is different, but maybe for the intel nuc bios since those are widely used and we both have them)?

It has been a couple of years since I last saw a report about problems with USB 3.2 hubs when used with the 400 Series cameras, so it does not seem like a common occurrence. Having said that, people do not typically say whether their hub is 3.1 or 3.2 when discussing problems with their camera on a hub connection and they are not asked what type it is, so there could have been numerous such cases in the past without knowing it.

Improvements to how librealsense handles multicam in the rs2::pipeline were introduced in SDK 2.35.2, though the improvements may not make a difference in cases where the problem is related to a particular USB hub model.

On the original RealSense F200 model in 2014, there was a RealSense user with some electronics knowledge who spliced a voltmeter into its USB cable to check the power draw in real-time. This was an expensive and risky project at that time because the cable was hard-wired into the camera. It may be cheaper to do now in the era of detachable USB cables (especially if done with an unofficial cable instead of the official one supplied with the camera so you still have the original as a backup).

The ideal USB BIOS configuration is probably "whatever the computer is using by default", since the cameras are designed to work with devices with any Intel or Arm processor (a small number of people have also used the camera successfully with AMD Ryzen and Threadripper processors). It is virtually unheard of for the USB settings to have to be adjusted in the BIOS, though a minority of very technically minded users have resorted to that in search of solutions to their problems. I will check my own USB BIOS settings on my NUC as a reference for you.

Here is my NUC's BIOS in the USB section. It does not reveal much, other than that Legacy USB Support is set to Enabled.

Ok, this is all useful. It seems like I should leave most of this stuff be for now.

Running with the timed offset using bash scripts (running in tmux) seems to be working well at 1080x720 on the color.

So I am exploring that a bit more	trying	outcome
reboot, offset starts at 1920x1080, initial reset true	one camera works, the other does not
reboot, offset starts at 1920x1080, initial reset false	one camera crashes, the other errors
reboot offset starts at 1080x720, initial reset false	one camera works, the other can't be found
reboot offset starts at 1080x720, initial reset true	one camera works, the other can't be found
continuing, unplug and plug back in the un found camera	camera is found, but doesn't publish any topics.
reboot offset starts at 1080x720, initial reset true	one camera works, one does not
continuing, kill and restart bad camera	good camera looses itself, but recovers. Failing camera continues failing
reboot offset starts at 1080x720, initial reset true	same one camera as the last two tests works and same one that doesn't does not. Makes me think that rebooting isn't resetting to the degree that I think it is and a prior test messed something up
shutdown, unplug bad camera, offset starts at 1080x720, initial reset true	both cameras work
reboot, offset starts at 1080x720, initial reset true	both cameras work
reboot, offset starts at 1080x720, initial reset true	both cameras work
reboot, offset starts at 1920x1080, initial reset true	one camera works, one does not (again flipped from earlier, no real pattern on what does and does not work)
shutdown, unplug & replug both imagers, offset starts at 1080x720, initial reset true	both work
reboot, offset starts at 1080x720, initial reset true	both cameras work
reboot, offset starts at 1080x720, initial reset true

At this point, I realize that 1080x720 is not really a real resolution, facepalm. I had only been testing whether the depth was actually working, not looking at the color. So when it is has been working, that is likely because color isn't even running. I assume this has been true for prior attempts... So let's take another look.

Using depth at 1280x720 because the tuning guide says to. I won't be changing this.
Want to use 1920x1080 for color
Will also try 1280x720 for color

trying	outcome
shutdown, unplug & replug both imagers, offset starts at 1920x1080 color, initial reset true	one camera works, the other seems to work, but doesn't publish anything (See Out 1)
shutdown, unplug & replug both imagers, offset starts at 1920x1080 color, initial reset true	same as prior
shutdown, unplug & replug both imagers, offset starts at 1280x720 color, initial reset true	both color and depth imagers work
shutdown and turn on, offset starts at 1280x720 color, initial reset true	both color and depth imagers work
reboot, offset starts at 1280x720 color, initial reset true	both color and depth imagers work
reboot, offset starts at 1280x720 color, initial reset true	both color and depth imagers work
shutdown, unplug & replug both imagers, offset starts at 1920x1080 color, initial reset true	one camera works one does not
reboot, offset starts at 1280x720 color, initial reset true	everything works
reboot, start each camera in its own pane in tmux waiting only on roscore at 1280x720 color, initial reset true	everything works, took a bit longer than expected to all come up
repeat above	one camera does not work, see Out 2
reboot, offset starts at 1280x720 color, initial reset true	everything works
reboot, offset starts at 1280x720 color, initial reset true	everything works
reboot, offset starts at 1280x720 color, initial reset false	both cameras fail, see Out 3 and Out 4
reboot, offset starts at 1280x720 color, initial reset true	Both cameras error out with "No RealSense devices were found!"
shutdown, unplug, replug, offset starts at 1280x720 color, initial reset true	everything works
reboot offset starts at 1280x720 color, initial reset true	everything works

So I think the takeaways are:

initial reset is needed. I can see from the IR projector that when I restart the computer with ROS running (which needs to work for my application) that the camera is staying on until the reboot cycle is over and the computer reboots, so definitely not shutting down cleanly as the computer goes down. Not sure how much that is ROS's fault or how much it is librealsense/librealsense-ros
the timing offset is needed. I don't know if this is a power thing or data thing or whatever.
1920x1080 on the color steams really doesn't seem to work.

I think that both of these things would be great to have built into realsense ROS. Have cameras come up staggered somehow. This is pretty easy to do in ROS 2, but in ROS 1 I am having to use bash to make that work, it is kind of messy. I think the initial reset option is a good tool, I am happy enough using that, the only downside is that it makes startup slower, but that is OK. The resolution thing has me confused.

Maybe I should take a look at just using the left imager with IR pattern removal on. That would mean that one less thing would be running, but should be the same amount of stuff transmitting. I'm not seeing a way to setup the color correction in realsense ros, is that doable?

Out 1

process[upper_realsense/realsense2_camera_manager-1]: started with pid [5400]                                                                                                                        [0/107]
process[upper_realsense/realsense2_camera-2]: started with pid [5401]
process[upper_realsense/color/image_throttled-3]: started with pid [5402]
process[upper_realsense/color/image_web-4]: started with pid [5404]
[ INFO] [1597876880.518014696]: Initializing nodelet with 8 worker threads.
[ INFO] [1597876880.575720934]: RealSense ROS v2.2.15
[ INFO] [1597876880.575744501]: Built with LibRealSense v2.36.0
[ INFO] [1597876880.575756180]: Running with LibRealSense v2.36.0
[ INFO] [1597876880.598095497]:  
[ INFO] [1597876880.995637328]: Device with serial number 912322061173 was found.

[ INFO] [1597876880.995670744]: Device with physical ID 2-3-2 was found.
[ INFO] [1597876880.995682094]: Device with name Intel RealSense D415 was found.
[ INFO] [1597876880.996362380]: Device with port number 2-3 was found.
[ INFO] [1597876880.996403123]: Resetting device...
[ INFO] [1597876887.088533606]:  
[ INFO] [1597876887.419412172]: Device with serial number 912322061173 was found.

[ INFO] [1597876887.419448060]: Device with physical ID 2-3-5 was found.
[ INFO] [1597876887.419463263]: Device with name Intel RealSense D415 was found.
[ INFO] [1597876887.419845704]: Device with port number 2-3 was found.
[ INFO] [1597876887.421625454]: getParameters...
[ INFO] [1597876887.485096808]: setupDevice...
[ INFO] [1597876887.485120506]: JSON file is not provided
[ INFO] [1597876887.485132361]: ROS Node Namespace: upper_realsense
[ INFO] [1597876887.485149574]: Device Name: Intel RealSense D415
[ INFO] [1597876887.485164305]: Device Serial No: 912322061173
[ INFO] [1597876887.485181633]: Device physical port: 2-3-5
[ INFO] [1597876887.485198773]: Device FW version: 05.12.06.00
[ INFO] [1597876887.485218371]: Device Product ID: 0x0AD3
[ INFO] [1597876887.485235984]: Enable PointCloud: Off
[ INFO] [1597876887.485249297]: Align Depth: On
[ INFO] [1597876887.485261728]: Sync Mode: On
[ INFO] [1597876887.485313305]: Device Sensors: 
[ INFO] [1597876887.529933327]: Stereo Module was found.
[ INFO] [1597876887.565178115]: RGB Camera was found.
[ INFO] [1597876887.565244630]: num_filters: 0
[ INFO] [1597876887.565256390]: Setting Dynamic reconfig parameters.
[ INFO] [1597876892.688553913]: Done Setting Dynamic reconfig parameters.
[ INFO] [1597876892.689562421]: depth stream is enabled - width: 1280, height: 720, fps: 30, Format: Z16
[ INFO] [1597876892.691757237]: color stream is enabled - width: 1920, height: 1080, fps: 30, Format: RGB8
[ INFO] [1597876892.694303754]: setupPublishers...
[ INFO] [1597876892.696699359]: Expected frequency for depth = 30.00000
[ INFO] [1597876892.723020784]: Expected frequency for color = 30.00000
[ INFO] [1597876892.744233280]: Expected frequency for aligned_depth_to_color = 30.00000
[ INFO] [1597876892.764107719]: setupStreams...
[ INFO] [1597876892.866338513]: insert Depth to Stereo Module
[ INFO] [1597876892.866396349]: insert Color to RGB Camera
 19/08 18:41:32,945 WARNING [140707672307456] (messenger-libusb.cpp:92) usb_request_queue returned error, endpoint: 132 error: Cannot allocate memory, number: 12
 19/08 18:41:33,078 WARNING [140707327239936] (messenger-libusb.cpp:92) usb_request_queue returned error, endpoint: 130 error: Cannot allocate memory, number: 12
[ INFO] [1597876893.089079223]: SELECTED BASE:Depth, 0
[ INFO] [1597876893.098936908]: RealSense Node Is Up!
 19/08 18:41:34,271 ERROR [140707672307456] (uvc-streamer.cpp:106) uvc streamer watchdog triggered on endpoint: 84

Out 2

[ WARN] [1597881333.441285672]: Device 2/2 failed with exception: set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
[ERROR] [1597881333.441319316]: The requested device with serial number 904412060717 is NOT found. Will Try again.
[ INFO] [1597881339.472116945]:  
 19/08 19:55:39,486 ERROR [140166836504320] (handle-libusb.h:95) failed to claim usb interface: 0, error: RS2_USB_STATUS_BUSY
 19/08 19:55:39,507 WARNING [140166950139648] (rs.cpp:304) null pointer passed for argument "device"
[ WARN] [1597881339.507505926]: Device 1/2 failed with exception: failed to set power state
 19/08 19:55:39,541 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:39,609 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:39,677 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:39,745 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:39,813 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:39,881 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:39,949 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,017 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,085 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,153 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,221 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,289 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,357 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,425 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,493 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,561 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,629 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,697 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,765 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,833 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,901 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:40,969 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,037 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,105 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,173 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,241 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,309 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,377 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,445 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,513 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,581 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,649 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,717 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,785 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,853 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,921 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:41,989 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 19:55:42,057 WARNING [140166819718912] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61

Out 3

19/08 20:06:37,580 WARNING [139998619760384] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:37,640 WARNING [139998619760384] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:37,648 WARNING [139998619760384] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:37,708 WARNING [139998619760384] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:37,715 WARNING [139998619760384] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:37,775 WARNING [139998619760384] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:37,783 WARNING [139998619760384] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:37,836 WARNING [139998724040448] (types.cpp:78) set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
[ERROR] [1597881997.837020404]: An exception has been thrown: set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
terminate called after throwing an instance of 'rs2::invalid_value_error'
  what():  set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
[upper_realsense/realsense2_camera_manager-1] process has died [pid 5549, exit code -6, cmd /opt/ros/kinetic/lib/nodelet/nodelet manager __name:=realsense2_camera_manager __log:=/home/nuc-admin/.ros/log/ea173b26-e278-11ea-a614-982cbccfb098/upper_realsense-realsense2_camera_manager-1.log].
log file: /home/nuc-admin/.ros/log/ea173b26-e278-11ea-a614-982cbccfb098/upper_realsense-realsense2_camera_manager-1*.log
[upper_realsense/realsense2_camera-2] process has finished cleanly
log file: /home/nuc-admin/.ros/log/ea173b26-e278-11ea-a614-982cbccfb098/upper_realsense-realsense2_camera-2*.log

Out 4

 19/08 20:06:27,682 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:27,718 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:27,750 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:27,785 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:27,817 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:27,853 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:27,885 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:27,920 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:27,952 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:27,988 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:28,020 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:28,055 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:28,087 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:28,123 WARNING [140124190729984] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 19/08 20:06:28,148 WARNING [140124215908096] (types.cpp:78) set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
[ERROR] [1597881988.157301983]: An exception has been thrown: set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
terminate called after throwing an instance of 'rs2::invalid_value_error'
  what():  set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable
[lower_realsense/realsense2_camera-2] process has finished cleanly
log file: /home/nuc-admin/.ros/log/ea173b26-e278-11ea-a614-982cbccfb098/lower_realsense-realsense2_camera-2*.log
[lower_realsense/realsense2_camera_manager-1] process has died [pid 4844, exit code -6, cmd /opt/ros/kinetic/lib/nodelet/nodelet manager __name:=realsense2_camera_manager __log:=/home/nuc-admin/.ros/log/ea173b26-e278-11ea-a614-982cbccfb098/lower_realsense-realsense2_camera_manager-1.log].
log file: /home/nuc-admin/.ros/log/ea173b26-e278-11ea-a614-982cbccfb098/lower_realsense-realsense2_camera_manager-1*.log

The RealSense ROS wrapper developer mentioned recently that official support for ROS2 is being worked on and is planned for a couple of months from the time of writing this.

https://github.com/IntelRealSense/librealsense/issues/5825#issuecomment-668829615

I note that in your logs, Align Depth is set to On. Is that required by your application, please? The documentation says about this function: "If set to true, will publish additional topics with all the images aligned to the depth image. The topics are of the form: /camera/aligned_depth_to_color/image_raw etc"

The projector that creates the dot pattern isn't generating a stream as it is a separate component from the IR imager, so having it on or off should not add or remove processing (though error over distance - RMS Error - can reduce by around 30% when the dot pattern is turned off). Laser Power is tied to the dot pattern though. As the Laser Power value is increased, the dot pattern becomes more visible, and becomes less visible if Laser Power is reduced.

A consequence of reducing Laser Power though is that the depth image can become more sparse, whilst increasing Laser Power makes the depth image less sparse.

On the SR300 camera models at least, laser power can affect the stability of the camera connection, with the possibility that the camera will disconnect once Laser Power is increased above a certain level. That issue does not tend to be seen on the 400 Series cameras though.

My understanding is that if you want to make adjustments in the ROS wrapper that are part of the camera's Advanced Mode then it has to be done by loading a json / visual preset file like the IR pattern removal one. The color correction matrix is an Advanced Mode function.

Yeah that is exciting.

I have Align Depth on because we are eventually going to be running some algorithms on the color and will want to look up the depth for points of interest. Maybe we will even go crazy at some point and operate directly on the 4D data. Although, now that you ask it, I see how that could place restrictions on the resolution which the imagers can run at. I had assumed that the color would be subsampled to generate the aligned image. Can you speak to that at all?

Right, but it is a good way to see if the cameras are powered up. If the laser is on, then the camera is powered up. So if the laser is still on after a reboot cycle, then I take that to imply that the camera didn't shut down cleanly while the computer (and in theory ROS) are shutting down.

Gotcha, we will take a look into using the JSON files.

Could you explain more please about what the term "4D data" means to you. Is it something like the "volumetric capture" that Intel has been using to capture the pixels of a scene from all angles and view from any angle?

https://www.youtube.com/watch?v=G0XUgnPl9KQ

The new RealSense D455 has enhanced support for depth to color alignment in its hardware design. Both depth and color imagers are the same FOV size, both are mounted internally on the same stiffener and for the first time the color imager has a fast global shutter like the depth imager does (it had a slower rolling shutter previously).

There is the complication that the D435 and D455 models do not support the projector pattern removal preset. The Chief Technical Officer of the RealSense Group at Intel (agrunnet) has previously highlighted an alternative preset file for that purpose, though I don't know if it works with non-D415 models.

https://github.com/IntelRealSense/librealsense/issues/6248#issuecomment-615902296

If you wanted to achieve a higher FPS without sacrificing resolution, Intel's External Synchronization (genlock) feature enables that on models with a global shutter such as the D435 models and D455. If you have multiple cameras, you can multiply an FPS by the number of cameras in a trigger sequence. So for example, if you had four D435 cameras running at 848x480 depth and 60 FPS, you could capture at 848x480 and 240 FPS (60 FPS x 4). As an extreme example, ten cameras running in 848x100 in "fast capture" 300 FPS mode could capture at 3000 FPS (300 x 10).

https://dev.intelrealsense.com/docs/external-synchronization-of-intel-realsense-depth-cameras

I'm not sure how the ROS wrapper handles librealsense, but in librealsense the depth FOV is aligned to the size of the color FOV. On D415 and D455, the color and depth FOV is the same.

A better way to check if the camera has shut down properly may be to check if the camera is busy in the program's start-up routine, as it should not be busy before pipe.start() is called.

https://github.com/IntelRealSense/librealsense/issues/2240

Nothing so fancy, just that we would process each pixel as having 4 channels instead of 3: RGBD. It isn't terribly important, just suffice to say that we do need color and depth aligned. Question for you: what is the ideal resolution for the color imager when we need it aligned to depth and the depth is at 1280x720?

the D455 looks very nice, but that min-z would not work for us. We are doing near field interactions with humans.

The higher fps is cool, not really relevant to our problem here though.

My recollection is that if you are doing depth to color alignment with the align processing block then the processing block will handle the math of doing the alignment based on whatever the color resolution is set to. So using 1920x1080 color should work out, assuming that it does not cause a crash (bearing in mind your earlier problems with 1920x1080 in this case).

Dorodnic has said that there is no strict dependency between RGB and depth.

https://github.com/IntelRealSense/librealsense/issues/3376#issuecomment-469345072

Ok, thanks. I think that I can make it work with what I have so far, so I am good with closing this issue. 2 things going forward:

I would love to have a better understanding of what is breaking here
Feature request: build a startup ordering component into realsense ros, essentially be able to give an index to each camera and each camera will come up in succession, waiting for the prior index to be set up before trying to start.

I wonder if the two cameras could be put on a mains electricity powered USB hub if they are not already, so that they are not dependent on the computer for their power. The hub could be turned off at the wall at the time when the computer is booted so that the cameras are not detected by the computer. After boot, the hub could then be turned on at the wall power socket and the cameras on the hub should receive power and be enumerated as USB devices. This could help to avoid device enumeration problems that might occur at boot-up.

Edit: I recalled that you are using a battery pack, so that idea may not be practical.

Could you post your feature request on a new issue please? Otherwise if an official feature request is filed, this entire existing discussion will have to be kept open whilst the feature request is being considered. Thanks!

Unfortunately the cameras are on a mobile platform with a battery. It is possible for us to put in some power circuity that would do that, but the added complexity is likely too high.

Put it in #1346

I re-reviewed the case. control_transfer returned errors are very hard to diagnose a specific cause for though. I recall a case where the errors were solved simply by updating OS and kernel. I note that your problems at the start of this case began when you updated the computer. So there may indeed by a causal relationship between OS / kernel and problems.

hmm, that is helpful to know. The OS is Ubuntu 16 with the latest kernel. Perhaps a simple re-install of the OS would fix the problem.

Could secure boot cause these sorts of problems?

There is not a great deal of concrete evidence for or against having secure boot turned off. Having Secure boot disabled was a recommendation for the previous RealSense ROS wrapper (realsense_camera) for the earlier generation of cameras though as it made DKMS packages fail. And having it enabled may affect patching of the video driver in Linux, in which case a member of the RealSense team recommended disabling it in 2019.

https://github.com/IntelRealSense/librealsense/issues/3354#issuecomment-469006598

If it is not a policy of your company that Secure Boot has to be enabled on its machines then it should be okay to turn it off.

Hi @mjsobrep Do you require further assistance with this case, please? Thanks!

Hey @MartyG-RealSense I think I am good for right now. I have some stuff that I want to check out, but have the information I think I need. Thanks!

Hi @mjsobrep Thanks very much for the update!

I am facing a similar issue. I have a robot with three D435i cameras and sometimes I got RS2_USB_STATUS_BUSY with any of them (it is not always the same camera which fails). Usually, I just stop and re-launch them again and everything works well. I added launch delays, but the error still appears. Actually, it appears the first time I try to run the cameras (when turning on the PC), although it appears any time. Cameras are directly connected to three different USB3 ports.

How to avoid this to happen?

Just for reference, I use RealSense ROS v2.2.20 and LibRealSense v2.40.0.

Thank you

Hi @jcgarciaca Which roslaunch method are you using to launch your multiple cameras please - rs_camera.launch or rs_multiple_devices.launch?

https://github.com/IntelRealSense/realsense-ros#work-with-multiple-cameras

Hi @MartyG-RealSense, actually I use rs_rgbd.launch with corresponding camera and serial_no arguments for every camera.

Are you putting each camera on a separate terminal on the same computer, with a separate roslaunch for each camera, as Step Two of Intel's multiple camera ROS tutorial suggests?

https://www.intelrealsense.com/how-to-multiple-camera-setup-with-ros/

Yes, you're right. But I use rs_rgbd

<terminal 1>
roslaunch realsense2_camera rs_rgbd.launch camera:=cam_1 serial_no:=<serial_number_1>
<2 sec delay>

<terminal 2>
roslaunch realsense2_camera rs_rgbd.launch camera:=cam_2 serial_no:=<serial_number_2>
<2 sec delay>

<terminal 3>
roslaunch realsense2_camera rs_rgbd.launch camera:=cam_3 serial_no:=<serial_number_3>

Do you experience any improvement in performance if you add a reset instruction to the end of each of the three roslaunch instructions so that the camera undergoes a hardware reset at launch (similar to a USB unplug-replug of the camera).

initial_reset:=true

I just tested with the initial_reset set to true, but I got the same error

@jcgarciaca You said in your original comment "Usually, I just stop and re-launch them again and everything works well". Could you tell me the method that you use to re-launch the cameras please?

In my application I have some sh scripts to execute roslaunch. Inside every sh file I have roslaunch realsense2_camera rs_rgbd.launch camera:=cam_X serial_no:=<serial_number_X> Then, when I need to stop the execution I do rosnode kill -a, killall -9 roscore and killall -9 rosmaster and I re-run the corresponding sh files and the cameras start properly.

Okay, thank you very much for the information. I will continue to investigate on Thursday. Thanks for your patience!

IntelRealSense / realsense-ros