Hardware sync not working perfectly on Jetson Xavier - Unable to set power state

[kun-faaya-kun]

---

Required Info
Camera Model	D435i
Firmware Version	5.15.0.2
Operating System & Version	Ubuntu 18.04
Kernel Version (Linux Only)	4.9.337-tegra
Platform	NVIDIA Jetson
SDK Version	2.54.1
Language	python
Segment	Robot

Issue Description

Hello, I am trying to sync multiple d435i cameras using the hardware sync.

From the hardware side, I have setup the circuitry as mentioned here. I am using USB3 cables, and my usb hub has external power supply as well.

From the software side, I am detecting and then sorting the serial numbers of the cameras connected in ascending order. I start the cameras with a second delay each. (https://dev.intelrealsense.com/docs/multiple-depth-cameras-configuration)

We also found it useful to start the cameras staggered sequentially in time with a few seconds delay, according to lowest serial number, as opposed to all at the same time

The first camera to start is my master, others are slaves. For master -

for sensor in device.query_sensors():
    if sensor.get_info(rs.camera_info.name) =="Stereo Module":
        sensor.set_option(rs.option.inter_cam_sync_mode, 1.0)
    device.first_depth_sensor().set_option(rs.option.inter_cam_sync_mode, 1.0)

For slave - (i have tried to set the mode to 2 as well for the slave)

for sensor in device.query_sensors():
    if sensor.get_info(rs.camera_info.name) =="Stereo Module":
        sensor.set_option(rs.option.inter_cam_sync_mode, 3.0)
    device.first_depth_sensor().set_option(rs.option.inter_cam_sync_mode, 3.0)

It is working perfectly on my laptop, but works say 70% of the times on my Jetson Xavier. The other times, it gives an error when i set the sync mode. Even if i set the sync mode through realsense-viewer, and try to normally start the pipeline, i get the same error about 30% of the times

Unable to set power state

When I connect 3 cameras together, the issue is more profound with atleast one of the cameras not starting most of the times (works perfectly on my laptop though)

I have also tried connecting the 2/3 cameras directly to the usb port of the jetson instead of using the usb hub, it did not change anything.

I would be very grateful for any help to troubleshoot this on the jetson, such that it works seemlessly every time. Thank you so much for your time! @MartyG-RealSense

[MartyG-RealSense]

Hi @kun-faaya-kun The first aspect of your project that I would recommend changing is using first_depth_sensor(). What this will do is pick one camera out of all that are attached and access its depth sensor. This means that each time the script is started, it will pick a random camera.

This runs the risk that you could set one particular camera as the master and then when using first_depth_sensor() again for the slaves, it may randomly choose the same camera that has already been used for the master and be unable to access it, causing a Failed to set power state error due to that access failure.

It would therefore be better to access individual cameras by their serial number so that you can be certain that the same camera will never be selected twice.

[kun-faaya-kun]

Hi @MartyG-RealSense, thank you for the reply! I am making sure that device is initialized with the right camera based on serial number.

device = None
ctx = rs.context()
devices = ctx.query_devices()
for dev in devices:
    if dev.get_info(rs.camera_info.serial_number) == serial_number:
        device = dev

Now for this particular device i set the inter cam sync mode. I do this for every serial number in a different process each.

I even tried commenting out the first_depth_sensor() line, but same results on the jetson, it works perfectly half the times, other times either one of the camera does not start, while the other records. I also noticed that it does not matter whether the camera is being set to slave or master when it fails.

Would be grateful for further guidance, thank you!

[kun-faaya-kun]

@MartyG-RealSense would be grateful for your guidance!

[MartyG-RealSense]

Hi @kun-faaya-kun Apologies for the delay in responding further. Do you still need guidance for this issue, please? Thanks!

[kun-faaya-kun]

Hi @MartyG-RealSense, yes the issue is still not resolved. I am unable to properly HW sync frames on my jetson xavier NX. My code works fine on my laptop. The issue is the same as mentioned above.

Thank you for your help!

[MartyG-RealSense]

On a past Failed to set power state case on Jetson involving multiple cameras, a workaround created by a RealSense user at https://github.com/IntelRealSense/librealsense/issues/5828#issuecomment-625671835 was to carefully control when each camera was started by using its serial number. Their scripting is in C++ but might give you some ideas.

[MartyG-RealSense]

Hi @kun-faaya-kun Do you require further assistance with this case, please? Thanks!

[kun-faaya-kun]

Hi @MartyG-RealSense , yes, I tried sorting the camera by serial number and then starting them sequentially. I also tried changing the order of the wires in the usb hub. None of it helped, still facing Failed to set power state on jetson xavier.

Thank you!

[MartyG-RealSense]

Did you build the librealsense SDK from source code on Jetson with CMake using the -DFORCE_RSUSB_BACKEND=TRUE flag in the CMake build instruction, please?

If RSUSB = true then the SDK will be slower to check the status of options than a kernel-patched build of the SDK, which can poll for events frequently to check for changes. RSUSB does not have that mechanism and so polls the device every 5 seconds by default. This means that when setting options, a change request may sometimes fail and have to be re-tried

So using a set_option instruction could result in an error such as Failed to set power state because the SDK was not ready to respond at that particular moment. https://github.com/IntelRealSense/librealsense/issues/6952 is an example of this problem.

[kun-faaya-kun]

Yes, indeed i built librealsense using -DFORCE_RSUSB_BACKEND=TRUE . Is there a workaround for it or should I rebuild it from source using -DFORCE_RSUSB_BACKEND=FALSE ?

Also, I have tried catching the Failed to set power state error and setting the cam sync mode again until it succeeds with a little sleep in between. That did not help, it goes into an infinite loop.

[MartyG-RealSense]

There is not a workaround for it so yes, I recommend rebuilding from source using -DFORCE_RSUSB_BACKEND=FALSE

[MartyG-RealSense]

Hi @kun-faaya-kun Do you require further assisstance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.

[kun-faaya-kun]

Hi @MartyG-RealSense , sorry for the delayed response. I tried building librealsense with RSUSB backend turned OFF and it worked. No more errors on Xavier now. Thank you so much!

[MartyG-RealSense]

It's no problem at all. I'm pleased to hear that you achieved a successful outcome with RSUSB = false. Thanks very much for the update!