Compressed depth topic gives 8 bit values

Camilochiang commented 2 years ago

Platform : Rpi4 + Ubuntu 18.04 Camera : D415 rosversion: 1.14.11 rosdistro: melodic rosversion realsense2_camera : 2.3.1

After recording a rosbag, the topic /camera/depth/image_rect_raw/compressed give only values up to 256 when using the following code for decompress in python but this is not the case with uncompressed images (with its normal 16 bits)

import rosbag
import cv2
from cv_bridge import CvBridge

bag_file = "2021-10-26-09-37-36.bag"
topic = '/camera/depth/image_rect_raw/compressed'

count = 0
for topic, msg, t in bag.read_messages(topics = [topic]):
    # Compressed at 8 bits / color
    depth_image = bridge.compressed_imgmsg_to_cv2(msg)
    print(depth_image)
    depth_array = np.array(depth_image, dtype=np.uint8)
    print(np.max(depth_array))

All the examples that i found were done with uncompressed data and the only issue that works with compressed data, is also working with 8 bits

MartyG-RealSense commented 2 years ago

Hi @Camilochiang It may be useful to compare your Python decompression code to a decompression Python node script written by another RealSense ROS user in https://github.com/IntelRealSense/realsense-ros/issues/1512 that utilizes the /camera/depth/image_rect_raw/compressed topic.

Like yourself, that script is using the bridge.compressed_imgmsg_to_cv2() instruction but additionally defining bgr8 in the brackets of the instruction.

Camilochiang commented 2 years ago

Hei @MartyG-RealSense ! thanks for your answer but that doesn't solve the problem :). I guess that is either a bug or my faul .. I manage to get the colorized (8 bits as is explained in README.me), soto keep going I'm following the tutorial that you have mention in a several times but just curious : is it possible to configure is_disparity from a ROS launch file? Does that variable exist in the actual software? I manage already to change the laser power and turn off histogram_equalization_enabled with the following code, but i don't get the inverse colorization as i would expect....

  <rosparam>
      /camera/stereo_module/laser_power: 300
      /camera/colorizer/histogram_equalization_enabled: false
      /camera/colorizer/is_disparity: true
  </rosparam>

Thanks!

MartyG-RealSense commented 2 years ago

If you are using Python then I would recommend not using the depth compression by colorization white-paper because of the difficulties involved in performing the depth recovery process outside of the RealSense SDK and in creating a Python conversion of the paper's recovery logic.

There is a long and detailed discussion with scripting about Python compression and decompression in https://github.com/IntelRealSense/librealsense/issues/8117 that might provide helpful insights.

It looks as though in the colorization paper, is_disparity is simply a bool for switching between applying depth to disparity or disparity to depth. One should be true and one false. Both should not be false or true. A Python script in the link below provides an example of setting True and False statuses for the two with Python code.

https://support.intelrealsense.com/hc/en-us/community/posts/360037286634-Strange-FPS-when-enable-both-color-and-depth-streams

It is possible to define a disparity filter in a ROS launch. It could be included in the roslaunch instruction with the 'disparity' filter definition, like this:

roslaunch realsense2_camera rs_camera.launch filters:=disparity

https://github.com/IntelRealSense/realsense-ros/issues/501 provides an example of defining filters in a launch file.

To get a list of available parameters for the disparity filter, please try running:

rosrun dynamic_reconfigure dynparam get /camera/disparity

Camilochiang commented 2 years ago

Hei @MartyG-RealSense once again for your help. I would comment some problems related to the mentioned:

The python implementation for transforming from compressed pictures to depth pictures is actually no so hard and it work quite smooth!. The problem is that the implementation presented in #9713 have a big problem: It consider r, g, b as bit8 values what make the full code fail, as their additions cannot be bigger than 255. For improvement if somebody want to use it just add int() to each channel . In any case the results are not the best ones, what is well indicated in the paper, specially for dynamic environments (Where the max depth will be constantly changing)
The problem persist: The compressed topic /camera/depth/image_rect_raw/compressed give a 8 bit picture instead of 16 bit. Any way to check it up? Also what is the topic /camera/depth/image_rect_raw/compressedDepth publishing? Is also 8 bits....

In another topic,, as I already have your attention: What is the function of enable_infra1 and enable_infra2 ? I could not find any documentation about it. All what I found was the note about infra_rgb on README.md

Thanks!

MartyG-RealSense commented 2 years ago

enable_infra1 and enable_infra2 enable the publishing of the topics for the left and right infrared streams respectively, which are disabled by default in the RealSense ROS wrapper.

In regard to your compression question, I will refer that to Doronhi the RealSense ROS wrapper developer. Thanks very much for your patience.

doronhi commented 2 years ago

Hi, Regarding the decompression of 16bit image, I managed to reproduce without the rosbag. While listening on "/camera/depth/image_rect_raw/compressed" and getting the message into "res":

bridge = CvBridge()
depth_image = bridge.compressed_imgmsg_to_cv2(res)
res.format  %yields: '16UC1; jpeg compressed '
depth_image.dtype   %yields: dtype('uint8')

I think this error is already dealt with here It is not strictly related to realsense-ros and although the PR was merged 20 days ago I don't know when the fix will be released.

Camilochiang commented 2 years ago

Hei @doronhi Thanks! I just must mention that /camera/color/image_raw/compressed is also giving me a 8 bit variable

>>> for topic, msg, t in bag.read_messages():
...     print(topic)
/camera/color/image_raw/compressed
>>> dir(msg)
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_check_types', '_connection_header', '_full_text', '_get_types', '_has_header', '_md5sum', '_slot_types', '_spec', '_type', 'data', 'deserialize', 'deserialize_numpy', 'format', 'header', 'serialize', 'serialize_numpy']
>>> msg.format
'rgb8; jpeg compressed bgr8'

Any information regarding about this issue?

doronhi commented 2 years ago

The color images are 3 channel 8-bit images. What is unexpected with 'rgb8; jpeg compressed bgr8' for topic '/camera/color/image_raw/compressed'? Am I missing something?

Camilochiang commented 2 years ago

Hei @doronhi . Please correct me if I'm wrong. The official Datasheet for the Intel Realsense product family D400 series (Revion 009 - June 2020) says that the picture should be YUY2 16 bits for the color Stream of the RGB camera (in my case 415). Isn't that the case? My apologies if is me who is confuse .

doronhi commented 2 years ago

No need to apologize. The camera's format is converted to RGB8 using librealsense2 functions to follow the ros2 standard sensor_msgs::image_encodings::RGB8. We could make realsense-ros send the color topic in other formats but I don't remember that the need ever came up.

Camilochiang commented 2 years ago

ah, OK. I don't think that is the case, as ROS is designed for robotic applications and ML have demonstrated so far to work better in 8 bits that 16 bits. Thanks for your help!

Camilochiang commented 2 years ago

@MartyG-RealSense I actually realize now, looking to some files that you never answer one question. What is the topic compressedDepth publishing it? Thanks!

MartyG-RealSense commented 2 years ago

Hi @Camilochiang I emphasize upfront that I am not certain about this particular question. My research suggests to me that compressedDepth is publishing depth data in a compressed state through compressed_depth_image_transport, as described at https://github.com/IntelRealSense/realsense-ros/issues/369 and https://github.com/IntelRealSense/realsense-ros/issues/1672

Camilochiang commented 2 years ago

Thanks @MartyG-RealSense ! Not 100% sure about the difference between compress and compressDepth topic but i will investigate it

in the other hand @doronhi, I did modify manualy cv_bridge and now is working (so python return 16bit from compressed msg). I confirm using the code from the PR. But still, when I decompress using depth_image = CvBridge().compressed_imgmsg_to_cv2(msg) i get 8 bit values and using depth_image = CvBridge().compressed_imgmsg_to_cv2(msg, desired_encoding="bgr16") I get wrong values (like 65 m, meanwhiles my max distance was 7m). Could you have a look to see what encoding should be use? With this last options all values are either close to 65m or 0, giving me a "monochromatic" image. ``

MartyG-RealSense commented 2 years ago

@Camilochiang Somebody at https://github.com/ros-perception/vision_opencv/issues/206 experienced the same issue with compressed_imgmsg_to_cv2(msg) returning an 8-bit image.

MartyG-RealSense commented 2 years ago

Hi @Camilochiang Do you require further assistance with this case, please? Thanks!

Camilochiang commented 2 years ago

Hei @MartyG-RealSense yes :) Im still waiting for the answer of @doronhi . Im just not sure if the message is really arriving as an 8 bit or I dont know how to decompress it ;)

MartyG-RealSense commented 2 years ago

Would using BRG8 for desired_encoding instead of BGR16 resolve your problem, please?

depth_image = CvBridge().compressed_imgmsg_to_cv2(msg, desired_encoding="bgr8")

Camilochiang commented 2 years ago

@MartyG-RealSense , I did try with all encoding (bgr8, bgr16, passthrough) but after decompressing the image I only get values really high values (65k) or 0 as I told you before. I will retry on Monday and let you know. Thanks for your help!

MartyG-RealSense commented 2 years ago

You are very welcome. I will be happy to continue providing support in your case. Doronhi is no longer working with RealSense ROS and the focus of RealSense ROS wrapper development is now on the ROS2 4.x wrapper, with further updates not planned for the ROS1 wrapper.

MartyG-RealSense commented 2 years ago

Hi @Camilochiang Do you require further assistance with this case, please? Thanks!

Camilochiang commented 2 years ago

I keep working on it. To avoid the usage of cv_bridge I tried with imageio (To read as bytes), but the compressed ros message keep coming out as a 8bits value, what make useless compresion in depth images.

image = io.BytesIO(last_msg.data)
iio.imread(image)
Array([[  0,   0,   0, ...,   0,   0,   0],
       [  0,   0,   0, ...,   0,   0,   0],
       [  0,   0,   0, ...,   0,   0,   0],
       ...,
       [  0,   0,   0, ..., 255, 255,   0],
       [  0,   0,   0, ..., 247, 254,   0],
       [  0,   0,   0, ...,   0,   0,   5]], dtype=uint8)
# Lets corroborate it that 16 bits works with imageio
 im1 = np.zeros((3,3), np.uint16)
 im1[0,0] = 1000
iio.imsave('test.png', im1)
 im2 = iio.imread('test.png')
 print(im1.dtype, im1[0, 0])  # -> uint16 1000
uint16 1000
print(im2.dtype, im2[0, 0])  # -> uint16 1000
uint16 1000

MartyG-RealSense commented 2 years ago

I conducted further research on the subject. The official RealSense SDK ROS documentation link below illustrates the difference in published topics when using compressed-image-transport and compressed-image-depth-transport

https://dev.intelrealsense.com/docs/ros-wrapper#section-compression-packages

I also located a past RealSense case with a similar situation to your own, where someone was attempting to obtain a 16UC1 image from compression and getting an 8-bit 8UC1 image instead.

https://community.intel.com/t5/Items-with-no-label/Realsense-ROS-Decompressing-compressedImages-to-get-original/m-p/717675

The above discussed linked to this one:

https://answers.ros.org/question/276352/how-to-understand-data-from-the-image_rawcompressed-topic/

MartyG-RealSense commented 2 years ago

Hi @Camilochiang Do you require further assistance with this case, please? Thanks!

Camilochiang commented 2 years ago

Hei @MartyG-RealSense . I tried another two approach and either I dont know how to recover the images as 16 bits or there is a bug

using bytes works only when set to uint8 (depth is a message coming from a rosbag)

>>> depth.format
'16UC1; jpeg compressed '
>>> depth = np.frombuffer(depth.data, dtype=np.uint16).reshape(720, 1280, -1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: cannot reshape array of size 114409 into shape (720,1280,newaxis)
>>> depth = np.frombuffer(depth.data, dtype=np.uint8).reshape(720, 1280, -1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: cannot reshape array of size 228818 into shape (720,1280,newaxis)
>>> depth = io.BytesIO(depth.data)

using image transport and reading writing a subscriber in python (test is a message from a rosbag)

# in terminal: rosrun image_transport republish compressed in:=/camera/depth/image_rect_raw out:=/test
# After starting the subscriber...
>>> np.frombuffer(test.msg.data, dtype=np.uint16).reshape(720, 1280)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: cannot reshape array of size 460800 into shape (720,1280)
>>> np.frombuffer(test.msg.data, dtype=np.uint8).reshape(720, 1280)
array([[0, 0, 0, ..., 0, 0, 0],
   [0, 0, 0, ..., 0, 0, 0],
   [0, 0, 0, ..., 0, 0, 0],
   ...,
   [0, 0, 0, ..., 0, 0, 0],
   [0, 0, 0, ..., 0, 0, 0],
   [0, 0, 0, ..., 0, 0, 0]], dtype=uint8)

In both cases only 8 bit are working. Is there any way to check if this is a bug or not? Greetings

MartyG-RealSense commented 2 years ago

I do not have any further advice that I can offer on this subject, unfortunately. If there was a bug in the ROS wrapper then there would not be a way to resolve it as the RealSense ROS wrapper development team are now focusing on the ROS2 4.x series of wrappers and the ROS1 2.x wrapper series will not receive further updates.

Camilochiang commented 2 years ago

Ok. Closed without a satisfactory answer. Thanks in any case!

Zhuomin-Zhou commented 2 years ago

I do not have any further advice that I can offer on this subject, unfortunately. If there was a bug in the ROS wrapper then there would not be a way to resolve it as the RealSense ROS wrapper development team are now focusing on the ROS2 4.x series of wrappers and the ROS1 2.x wrapper series will not receive further updates.

Hi,I also met the issue when I was using the ros2 wrapper for getting the compressed depth frame(D435, Foxy, cv_bridge, Python). Is there any better way to solve that ?

MartyG-RealSense commented 2 years ago

Hi @Zhuomin-Zhou At the bottom of the page linked to below - in the section titled Image transport problems and compressed topics - it is reported that there have been problems for ROS2 Foxy users in general (not the RealSense ROS2 wrapper specifically) with the ROS2 image_transport_plugins package, with links provided to example cases about the problem.

https://www.stereolabs.com/docs/ros2/

In the links that the above page recommends, a ROS2 Foxy user suggested a method for fixing the issue at https://github.com/stereolabs/zed-ros2-wrapper/issues/31#issuecomment-953447841

Zhuomin-Zhou commented 2 years ago

Hi @Zhuomin-Zhou At the bottom of the page linked to below - in the section titled Image transport problems and compressed topics - it is reported that there have been problems for ROS2 Foxy users in general (not the RealSense ROS2 wrapper specifically) with the ROS2 image_transport_plugins package, with links provided to example cases about the problem.

https://www.stereolabs.com/docs/ros2/

In the links that the above page recommends, a ROS2 Foxy user suggested a method for fixing the issue at https://github.com/stereolabs/zed-ros2-wrapper/issues/31#issuecomment-953447841

Thank you for your help!!

IntelRealSense / realsense-ros

Compressed depth topic gives 8 bit values #2140