D455 on Jetson Xavier very slow

[GEngels]

I am trying to use the Intel Realsense camera on the Jetson Xavier AGX. When I only run the RGB camera & depth image it seems to run quite fast (30fps). However, when I enable the pointcloud it becomes very slow.

I tried to run it at 15 fps for RGB & Depth at various resolutions but the result is the same. The RGB camera freezes and no point cloud is displayed in RVIZ. When I block part of the depth camera (so that it doesn't have to create as many points in the point cloud) it seems to work again. The creation of the pointcloud in the realsense node seems to be a bottleneck, I am running the Jetson on max performance.

Some of the errors and warnings that appear are the following:

"incomplete frame received: Incomplete video frame detected! Size 99080 out of 2048255 bytes (4%):" "Hardware Notification:Depth stream start failure,1.65668e+12,Error,Hardware Error" "Out of frame resources"

I wonder if it is expected behaviour that you can only run it with maybe 5 FPS on the Jetson on max settings.

[iraadit]

Hi @MartyG-RealSense,

I finally found a way to get a (nearly) 30FPS textured pointcloud, but it is suboptimal. It consists of these different elements:

1. 1280x720x30 for depth and color
2. decimation_filter.enable:=true decimation_filter.filter_magnitude:=3 (kinda work with :=2 (default))
3. json_file_path:=ABSOLUTE_PATH_TO/HighAccuracyPreset.json
4. depth_module.enable_auto_exposure:=false depth_module.exposure:=32000
5. OpenGL implementation of _get_texturemap
6. Using the script _topichz.py

With that, I can get nearly 30FPS (more than 25) on color, depth and pointcloud. But it is at the expense of a lower resolution of the pointcloud (due to use of the High Accuracy preset followed by the decimation filter).

topic_hz.py

With rviz2, I could see the streams framerates dropping when I was using ros2 topic hz. Searching for it on internet, I found this post describing the same thing, started to write my own suscriber to calculate the FPS and then discovered it had already been done on realsense-ros repo. For reference, the script is here: realsense2_camera/scripts/topic_hz.py. Using this script, I can (finally) get good framerates values. The script is only present in the ros2-beta branch (as it seems it wasn't a problem with ROS 1).

Depth module

32000 instead of 33000 or auto (because then saturate at 15FPS).

For the exposure of the depth module, would there be something similar to _auto_exposurepriority for the color stream?

Preset

I got the preset from the page D400-Series-Visual-Presets wiki on the librealsense repo.

Decimation

On this wiki page, it is also said:

The key to good depth is to start from our recommended defaults:

D435: Use 848x480 resolution @30fps, with auto-exposure. Use post processing with downsample 2.
D415: Use 1280x720 resolution @30fps, with auto-exposure. Use post processing with downsample 3.

What would be the recommended default for D455?

Concerning decimation, I also saw on the web documentation of intelrealsense that there would be a better implementation of decimation for collision avoidance problems. I'll add it in our code.

Decimation

Downsample Step

Downsampling is a very common first step in any depth processing algorithm. The key observation is that downsampling reduces spatial (X-Y) accuracy but preserves Z-accuracy. It is so widespread that the SDK offers built-in downsampling method in form of rs2::decimation_filter. It's important to note that using standard OpenCV downsampling is not ideal for depth images. In this example we show another correct way to implement depth downsampling. It is conceptually similar to rs2::decimation_filter, picking one of the non-zero depth values for every 4x4 block, but unlike rs2::decimation_filter it is picking the closest depth value instead of median value. This makes sense in context of collision avoidance, since we want to preserve the minimal distance to an object.

Naive implementation for this approach:

C++

for (int y = 0; y < sizeYresized; y++)
        for (int x = 0; x < source.cols; x += DOWNSAMPLE_FACTOR)
        {
            uint16_t min_value = MAX_DEPTH;

            // Loop over 4x4 quad
            for (int i = 0; i < DOWNSAMPLE_FACTOR; i++)
                for (int j = 0; j < DOWNSAMPLE_FACTOR; j++)
                {
                    auto pixel = source.at<uint16_t>(y * DOWNSAMPLE_FACTOR + i, x + j);
                    // Only include non-zero pixels in min calculation
                    if (pixel) min_value = std::min(min_value, pixel);
                }

            // If no non-zero pixels were found, mark the output as zero
            if (min_value == MAX_DEPTH) min_value = 0;

            pDest->at<uint16_t>(y, x / DOWNSAMPLE_FACTOR) = min_value;
        }

OpenGL

I also have a modification of the code to execute part of the pointcloud calculation with OpenGL. I should test without this modification to see if I still get the nearly 30 FPS. (CPU usage is for sure down with this modification, as well as pointcloud processing is faster).

Conclusion

Can you help me with any of the questions I have?

I will finally have to let this case at rest in the coming days to work on other things. I'm still not satisfied with the result and it can be said for sure that the combination of D455, ROS 2 (Humble - Isaac), librealsense 2.51.1, realsense-ros (ros2-beta) and trying to get a textured pointcloud with the highest resolution possible on a Jetson AGX Xavier (Jetpack 4 or 5) is giving subpar results for now.

I will submit some Pull Requests in the coming days.

Thank you for your help

[MartyG-RealSense]

Thanks so much @iraadit for sharing such detailed feedback of your tests!

Using the Medium Density preset - MedDensityPreset.json - may provide a better image than High Accuracy, as HA tends to greatly reduce the amount of detail on the depth image due to confidence-filtering of depth coordinates, whilst Medium Density provides a good balance between accuracy and the amount of detail on the image.

Regarding a similar option to the RGB-only auto_exposure_priority for the depth stream: when auto-exposure is disabled then a constant FPS can be enforced if the manual epxosure value is within a certain range, as mentioned earlier in this discussion at https://github.com/IntelRealSense/realsense-ros/issues/2396#issuecomment-1231663581 but there is not a direct depth-stream equivalent for auto_exposue_priority

The recommended resolution setting for optimal depth accuracy on the D455 camera model is the same as those for D435 / D435i - 848x480 depth at 30 FPS.

[MartyG-RealSense]

Does anyone who commented on this case require further assistance, please? Thanks!

[AndreV84]

not from my side. thanks for following up though

[MartyG-RealSense]

Thanks very much, @AndreV84 :)

[AndreV84]

@MartyG-RealSense do you know by any chance a framework/model/ example which will detect 3d objects from recorded by realsense pointcloud files? Thanks

[MartyG-RealSense]

@AndreV84 If you mean using .ply pointcloud files then PyTorch3D might be a suitable option for object detection if you are able to use Python.

PyTorch3D https://pytorch3d.org/

https://ai.facebook.com/blog/building-3d-deep-learning-models-with-pytorch3d/

Loading ply files into PyTorch3D https://pytorch3d.org/docs/meshes_io

[iraadit]

Hi @MartyG-RealSense

Sorry, I've been busy on something else for the last two weeks.

I'll work again on the realsense pointcloud optimization this week.

[MartyG-RealSense]

No problem at all, @iraadit - thanks very much for the update and good luck!

[MartyG-RealSense]

Hi @iraadit Do you have an update about this case that you can provide, please? Thanks!

[iraadit]

Hi @MartyG-RealSense, We decided to wrap up our work on the realsense optimization a moment ago, it went better thanks to the different tricks mentioned before, but there would certainly still be room to optimize more. Thank you for your help. You can close the ticket.

[MartyG-RealSense]

Thanks very much @iraadit for the update!

As you are happy to close the ticket and there have not been further comments from other RealSense users on this discussion, I will close it. Thanks again!