Unable to Decompress Images in ROS2 using Pylon SDK in Python (or without Pylon SDK)

Hello,

I am currently working on a project where I need to decompress images coming from a Basler camera in a ROS2 environment using the Pylon SDK in Python (or any other method that can achieve this goal) to visualize them in RVIZ2. The images are published to a ROS2 topic as sensor_msgs/Image messages, but they are actually compressed and appear as static noise when visualized in RVIZ2. My objective is to decompress these images and publish them again in a decompressed format that can be visualized correctly in RVIZ2.

Issue and Steps to Reproduce

I am using the following Python code to attempt to decompress the images using the Pylon SDK (still experminting):

#!/usr/bin/env python3

import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from pypylon import pylon
from cv_bridge import CvBridge
import numpy as np
import cv2

class PylonImageDecompressorNode(Node):
    def __init__(self):
        super().__init__('pylon_image_decompressor')

        self.bridge = CvBridge()

        # Set up the Pylon image decompressor
        self.decompressor = pylon.ImageDecompressor()

        # Subscriber to the compressed image topic (published as Image)
        self.image_sub = self.create_subscription(
            Image,  # Message type for the incoming image even if it is compressed
            '/my_camera/pylon_ros2_camera_node/image_raw',  # The actual topic with compressed data
            self.image_callback,
            10
        )

        # Publisher to output the decompressed image
        self.image_pub = self.create_publisher(
            Image,  # Message type: Image
            'pylon/camera/decompressed_image',  # Topic to publish to
            10
        )

    def image_callback(self, msg):
        self.get_logger().info('Received a compressed image wrapped in an Image message.')

        try:
            # Extract raw data from the ROS2 Image message
            np_arr = np.frombuffer(msg.data, np.uint8)

            # Calculate buffer size in bytes
            buffer_size = np_arr.size * np_arr.itemsize

            # Create a Pylon image object
            pylon_image = pylon.PylonImage()

            # Attach the raw buffer to the Pylon image object using the correct parameters
            pylon_image.AttachUserBuffer(
                np_arr.ctypes.data,  # Pointer to the data
                buffer_size,  # Size of the data in bytes
                pylon.PixelType_Mono8,  # EPixelType for mono8 format
                msg.width,  # Image width
                msg.height,  # Image height
                0,  # PaddingX (no padding in mono8 format)
                pylon.ImageOrientation_TopDown  # Image orientation (default TopDown)
            )

            # Decompress the image using the Pylon SDK
            decompressed_image = self.decompressor.DecompressImage(pylon_image)

            # Convert the decompressed Pylon image to a numpy array (OpenCV format)
            img_array = decompressed_image.GetArray()

            # Determine the correct encoding format
            if len(img_array.shape) == 2 or img_array.shape[2] == 1:
                img_array = cv2.cvtColor(img_array, cv2.COLOR_GRAY2BGR)
                encoding = "bgr8"
            else:
                encoding = "bgr8"

            # Convert the numpy array (OpenCV image) to a ROS Image message
            image_msg = self.bridge.cv2_to_imgmsg(img_array, encoding=encoding)

            # Publish the decompressed image to the ROS topic
            self.image_pub.publish(image_msg)

            self.get_logger().info("Published decompressed image.")

            # Release resources
            decompressed_image.Release()

        except Exception as e:
            self.get_logger().error(f"Error decompressing image: {e}")

    def destroy_node(self):
        super().destroy_node()

def main(args=None):
    rclpy.init(args=args)
    node = PylonImageDecompressorNode()
    rclpy.spin(node)
    node.destroy_node()
    rclpy.shutdown()

if __name__ == '__main__':
    main()

I have also used this to successfully decompress images directly from a connected camera (will not work with pylon_ros2_driver):


#!/usr/bin/env python3

import rclpy  
from rclpy.node import Node  
from sensor_msgs.msg import Image  
from pypylon import pylon  # Pylon SDK for Basler camera interaction
from cv_bridge import CvBridge  # Bridge between ROS 2 image messages and OpenCV
import numpy as np  # NumPy for array manipulation
import cv2  # OpenCV library for image processing

class PylonImageDecompressorNode(Node):
    def __init__(self):
        # Initialize the node with the name 'pylon_image_decompressor'
        super().__init__('pylon_image_decompressor')

        # Create a publisher to output the decompressed image
        self.image_pub = self.create_publisher(
            Image,  # Message type: Image
            'pylon/camera/decompressed_image',  # Topic to publish to
            10  # Queue size for outgoing messages
        )

        # Initialize CvBridge to convert between ROS and OpenCV images
        self.bridge = CvBridge()

        # Set up the Basler camera
        self.camera = self.setup_camera()

        # Set up the Pylon image decompressor
        self.decompressor = pylon.ImageDecompressor()

        # Retrieve and set the compression descriptor from the camera
        descriptor = self.camera.BslImageCompressionBCBDescriptor.GetAll()
        self.decompressor.SetCompressionDescriptor(descriptor)

        # Start grabbing images from the camera
        self.camera.StartGrabbing(pylon.GrabStrategy_LatestImageOnly)

        # Main loop for grabbing and processing images
        self.timer = self.create_timer(0.1, self.process_images)

    def setup_camera(self):
        # Get the first available camera device
        camera = pylon.InstantCamera(pylon.TlFactory.GetInstance().CreateFirstDevice())
        camera.Open()

        # Configure the camera settings
        camera.Width.SetValue(2600)  # Example width, set according to your camera specs
        camera.Height.SetValue(2128)  # Example height, set according to your camera specs
        camera.ImageCompressionMode.SetValue('BaslerCompressionBeyond')
        camera.ImageCompressionRateOption.SetValue('Lossless')
        camera.BslImageCompressionRatio.SetValue(90.0)

        self.get_logger().info(f"Camera {camera.GetDeviceInfo().GetModelName()} initialized with compression settings.")

        return camera

    def process_images(self):
        if self.camera.IsGrabbing():
            # Retrieve the latest image from the camera
            grab_result = self.camera.RetrieveResult(5000, pylon.TimeoutHandling_ThrowException)

            if grab_result.GrabSucceeded():
                try:
                    # Decompress the image
                    decompressed_image = self.decompressor.DecompressImage(grab_result)

                    # Convert the decompressed Pylon image to a numpy array (OpenCV format)
                    img_array = decompressed_image.GetArray()

                    # Check the number of channels in the image
                    if len(img_array.shape) == 2 or img_array.shape[2] == 1:
                        # Image is single channel (grayscale), convert to BGR format
                        img_array = cv2.cvtColor(img_array, cv2.COLOR_GRAY2BGR)
                        encoding = "bgr8"
                    else:
                        # Image is already in BGR format
                        encoding = "bgr8"

                    # Convert the numpy array (OpenCV image) to a ROS Image message
                    image_msg = self.bridge.cv2_to_imgmsg(img_array, encoding=encoding)

                    # Publish the decompressed image to the ROS topic
                    self.image_pub.publish(image_msg)

                    self.get_logger().info("Published decompressed image.")

                    # Release resources
                    decompressed_image.Release()
                    grab_result.Release()

                except Exception as e:
                    self.get_logger().error(f"Error decompressing image: {e}")
            else:
                self.get_logger().warn(f"Image grab failed: {grab_result.GetErrorDescription()}")

    def destroy_node(self):
        # Stop the camera grabbing when the node is destroyed
        if self.camera.IsGrabbing():
            self.camera.StopGrabbing()
        self.camera.Close()
        super().destroy_node()

def main(args=None):
    rclpy.init(args=args)
    node = PylonImageDecompressorNode()
    rclpy.spin(node)
    node.destroy_node()
    rclpy.shutdown()

if __name__ == '__main__':
    main()

Issue Encountered

I could not find a way to retrieve all the required data for the pylon decompressor (like Descriptor) considering the camera is already working using the ROS2 Driver and I can not open the camera or currently, I encountered the following error:

[INFO] [1725900784.144116477] [pylon_image_decompressor]: Received a compressed image wrapped in an Image message.
[ERROR] [1725900784.144999987] [pylon_image_decompressor]: Error decompressing image: Wrong number or type of arguments for overloaded function 'ImageDecompressor_DecompressImage'.
  Possible C/C++ prototypes are:
    Pylon::CImageDecompressor::DecompressImage(Pylon::CGrabResultPtr const &)
    Pylon::CImageDecompressor::DecompressImage(void const *,size_t)

This is the ros2 topic echo for /image_raw for the compressed images:

image_data.txt

Hardware setup description

Camera Model(s): a2A4200-12gcPRO
PC or Embedded System Model/Type: ASUS Vivobook
CPU Architecture: X86_64
Operating System: Ubuntu 22.04
RAM: 16 GB
Interfaces Used to Connect the Cameras:
- Interface Type: GigE

Runtime information

echo $ROS_DISTRO
humble

pylon info: 8.0.0.10 64-Bit pylon Setup info: 7.5.0.15658 pylon Application info: 3.3.0.15479 64-Bit pylon Viewer info: 7.5.0.15479 64-Bit

Host system infos: CPU architecture: x86_64 Kernel type: linux Kernel version: 6.8.0-40-generic OS type: ubuntu OS version: 22.04 Render Backend: OpenGL

Screen(s): Screen 0 scale factor: 1 Screen 0 resolution: 2880x1620

Window(s): pylon Viewer 64-Bit" on screen 0

Host network infos: Host network adapter enp45s0: Name: enp45s0 MAC address: 58:11:22:89:3B:E1 Host network adapter docker0: Name: docker0 MAC address: 02:42:9C:42:4B:72 IP address: 172.17.0.1 Subnet mask: 255.255.0.0 Host network adapter br-a296f01552f4: Name: br-a296f01552f4 MAC address: 02:42:36:9B:A9:47 IP address: 172.18.0.1 Subnet mask: 255.255.0.0 Host network adapter wlp46s0: Name: wlp46s0 MAC address: 14:13:33:88:A3:99 IP address: 192.168.0.212 Subnet mask: 255.255.255.0



### Is your camera operational with the Basler pylon Viewer on your platform?

Yes

basler / pylon-ros-camera