Depth image msg contains only zero values

[feendres]

Preliminary Checks

• [X] This issue is not a duplicate. Before opening a new issue, please search existing issues.
• [X] This issue is not a question, feature request, or anything other than a bug report directly related to this project.

Description

Hello everybody,

I have a ROS Service which when triggered tries to get a depth image message via: img_depth_msg_ = ros::topic::waitForMessage<sensor_msgs::Image>(image_depth_topic_, nh_, ros::Duration(3)) Then i test if this message is null and if not i convert it: cv::Mat temp_depth_img = cv_bridge::toCvShare(img_depth_msg_, img_depth_msg_->encoding)->image; This OpenCV mat object sometimes(randomly) only contains zeros.

In my first attempt to debug this i loop getting depth image msg until i get a cv::mat which has a nonzero value. This happens most of the times immediately with the next retrieved message (without restarting the ZED Wrapper.) But I also had the case that 3 iterations where not enough to get a "non-zero" msg.

I haven't checked the raw uint8[] data from the message. Since i am not sure how to correctly convert the data (big_endian and casting). But i would think that this is not a problem of the cv::bridge converting.

Steps to Reproduce

I cannot really say how to reproduce it since it is randomly happening. 1.Start ZED-ROS-Wrapper 2.Get depth image msg with ros::topic::waitForMessage ...

Expected Result

The Image should have non-zero values.

Actual Result

The Image contains only zero values.

ZED Camera model

ZED2i

Environment

I had this issue on
OS: Ubuntu 20.04
CPU: x86
GPU: NVIDIA T1200
SDK: 4.0.5
Additional Info: ROS-Noetic
and 
OS: L4T 35.1.2
CPU: ARM
GPU: Jeston Xavier AGX
SDK: 4.0.8
Additional Info: ROS-Noetic

Anything else?

This is my configuration in the common.yaml:

brightness: 4 # Dynamic contrast: 4 # Dynamic hue: 0 # Dynamic saturation: 4 # Dynamic sharpness: 4 # Dynamic gamma: 8 # Dynamic - Requires SDK >=v3.1 auto_exposure_gain: true # Dynamic gain: 100 # Dynamic - works only if auto_exposure_gain is false exposure: 100 # Dynamic - works only if auto_exposure_gain is false auto_whitebalance: true # Dynamic whitebalance_temperature: 42 # Dynamic - works only if auto_whitebalance is false depth_confidence: 50 # Dynamic depth_texture_conf: 100 # Dynamic point_cloud_freq: 3.0 # Dynamic - frequency of the pointcloud publishing (equal or less to grab_frame_rate value)

depth: depth_mode: 'ULTRA' # 'NONE', 'PERFORMANCE', 'QUALITY', 'ULTRA', 'NEURAL' depth_stabilization: 0 # [0-100] - 0: Disabled openni_depth_mode: true # 'false': 32bit float meter units, 'true': 16bit uchar millimeter units

pos tracking, mapping, etc. is false.

This is my zed2i.yaml

general: camera_model: 'zed2i' grab_resolution: 'HD1080' # 'HD2K', 'HD1080', 'HD720', 'VGA', 'AUTO' grab_frame_rate: 10 # Frequency of frame grabbing for internal SDK operations

depth: min_depth: 0.3 # Min: 0.3, Max: 3.0 - Default 0.7 - Note: reducing this value wil require more computational power and GPU memory max_depth: 3.0 # Max: 40.0

[Myzhar]

Hi @feendres the depth map contains 32-bit float values. You must convert it by taking this into account. Furthermore, it is expected that many values are zero, mostly at the beginning of the array.

Have you tried to visualize the image after normalizing it to [0,255]?

[feendres]

Hi @Myzhar I use this configuration: openni_depth_mode: true So i have uint16 values and i always check the encoding when converting via cv::bridge and this always is correct. I saved the image and checked in GIMP and all values where indeed zero.

[Myzhar]

Can you send a color picture of the environment that you are acquiring? Are there objects too close to the camera that invalidate the depth perception?

[feendres]

This is the RGB image and the according depth image. The thing is i don't change anything in the ernvironment and the camera is not moved and sometimes when i trigger the service i don't get a "valid" depth image.

[Myzhar]

Regarding the depth picture, it is not normalized, that's why it's only back/white, but depth values are present, so the ZED node is working as expected. I recommend you check the behaviors of the function toCvShare to be sure it's converting the depth image correctly

[feendres]

@Myzhar Sorry i misunderstood you. I thought you wanted a correct depth image to get an impression from the scene. Here is a depth image when i encountered the problem. That is a png image saved with cv::imwrite directly after converting to cv::Mat.

[Myzhar]

It is expected to be black. You must convert it to visualize: https://support.stereolabs.com/hc/en-us/articles/5365701074967-Why-is-the-depth-map-so-dark

[feendres]

It is not a visualization problem. This are the histograms of a correct depth image and the depth image i uploaded in the last post. All values are zero.

[feendres]

To check if also the data in the msg is all zeros can i use code like that?

cv::findNonZero(temp_depth_img, depth_non_zero_idx_);
if(depth_non_zero_idx_.size() == 0)
            {
                uint16_t* depths = (uint16_t*)(img_depth_msg_->data[0]);
                int count = img_depth_msg_->height * img_depth_msg_->width;
                for(int i = 0; i < count; i++)
                {
                    if(depths[i] > 0)
                    {
                        ROS_INFO_STREAM_COND(print_debug_, "depths[i]: " << depths[i]);
                    }
                }
            }

I used part of the code from here tutorials/zed_depth_sub_tutorial/src/zed_depth_sub_tutorial.cpp

I think when i run this code on the Jetson this is problematic because of ARM being littleEndian?!

[Myzhar]

Can you visualize the depth map in rviz or rqt?

[feendres]

Can you visualize the depth map in rviz or rqt?

Yeah, the ROS Wrapper seems to be working fine 99,9% of the time. I had just sometimes the mentioned issues when i try to get image messages as described above.

Could it be that the camera has some frame drops? The camera is connected via the official 5m usb cable, but i don't use the extra micro-usb port. Could that be the problem?

[Myzhar]

No, that should not be a problem. If there are frame drops, the data is not published. You surely have an issue when converting from different data formats

[feendres]

No, that should not be a problem. If there are frame drops, the data is not published. You surely have an issue when converting from different data formats

Ok, thanks for the information. I will try some further tests.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days