When doing object detection, why is the bit point cloud data information empty

[kay2019-art]

###########################################################################################################################

License: Apache 2.0. See LICENSE file in root directory.

###########################################################################################################################

Simple Box Dimensioner with multiple cameras: Main demo file

###########################################################################################################################

Workflow description:

1. Place the calibration chessboard object into the field of view of all the realsense cameras.

Update the chessboard parameters in the script in case a different size is chosen.

2. Start the program.

3. Allow calibration to occur and place the desired object ON the calibration object when the program asks for it.

Make sure that the object to be measured is not bigger than the calibration object in length and width.

4. The length, width and height of the bounding box of the object is then displayed in millimeters.

###########################################################################################################################

Import RealSense, OpenCV and NumPy

import pyrealsense2 as rs import cv2 import numpy as np

Import helper functions and classes written to wrap the RealSense, OpenCV and Kabsch Calibration usage

from collections import defaultdict from realsense_device_manager import DeviceManager from calibration_kabsch import PoseEstimation from helper_functions import get_boundary_corners_2D from measurement_task import calculate_boundingbox_points, calculate_cumulative_pointcloud, visualise_measurements

def run_demo():

# Define some constants
resolution_width = 1280 # pixels
resolution_height = 720 # pixels
frame_rate = 15  # fps

dispose_frames_for_stablisation = 30  # frames

chessboard_width = 9 # squares
chessboard_height = 6   # squares
square_size = 0.0253 # meters

try:
    # Enable the streams from all the intel realsense devices
    rs_config = rs.config()
    rs_config.enable_stream(rs.stream.depth, resolution_width, resolution_height, rs.format.z16, frame_rate)
    rs_config.enable_stream(rs.stream.infrared, 1, resolution_width, resolution_height, rs.format.y8, frame_rate)
    rs_config.enable_stream(rs.stream.color, resolution_width, resolution_height, rs.format.bgr8, frame_rate)

    # Use the device manager class to enable the devices and get the frames
    device_manager = DeviceManager(rs.context(), rs_config)
    device_manager.enable_all_devices()

    # Allow some frames for the auto-exposure controller to stablise
    for frame in range(dispose_frames_for_stablisation):
        frames = device_manager.poll_frames()

    assert( len(device_manager._available_devices) > 0 )
    """
    1: Calibration
    Calibrate all the available devices to the world co-ordinates.
    For this purpose, a chessboard printout for use with opencv based calibration process is needed.

    """
    # Get the intrinsics of the realsense device
    intrinsics_devices = device_manager.get_device_intrinsics(frames)
    # while True:
    #   frames_devices = device_manager.poll_frames()
    #
    #   for (device_info, frame) in frames_devices.items():
    #       device = device_info[0]  # serial number
    #       color_image = np.asarray(frame[rs.stream.color].get_data())
    #
    #       # Visualise the results
    #       cv2.imshow('Color image from RealSense Device Nr: ' + device, color_image)
    #       cv2.waitKey(1)

    # Set the chessboard parameters for calibration
    chessboard_params = [chessboard_height, chessboard_width, square_size]

    # Estimate the pose of the chessboard in the world coordinate using the Kabsch Method
    calibrated_device_count = 0
    while calibrated_device_count < len(device_manager._available_devices):
        frames = device_manager.poll_frames()
        pose_estimator = PoseEstimation(frames, intrinsics_devices, chessboard_params)
        transformation_result_kabsch  = pose_estimator.perform_pose_estimation()
        object_point = pose_estimator.get_chessboard_corners_in3d()
        calibrated_device_count = 0
        for device_info in device_manager._available_devices:
            device = device_info[0]
            if not transformation_result_kabsch[device][0]:
                print("Place the chessboard on the plane where the object needs to be detected..")
            else:
                calibrated_device_count += 1

    # Save the transformation object for all devices in an array to use for measurements
    transformation_devices={}
    chessboard_points_cumulative_3d = np.array([-1,-1,-1]).transpose()
    for device_info in device_manager._available_devices:
        device = device_info[0]
        transformation_devices[device] = transformation_result_kabsch[device][1].inverse()
        points3D = object_point[device][2][:,object_point[device][3]]
        points3D = transformation_devices[device].apply_transformation(points3D)
        chessboard_points_cumulative_3d = np.column_stack( (chessboard_points_cumulative_3d,points3D) )

    # Extract the bounds between which the object's dimensions are needed
    # It is necessary for this demo that the object's length and breath is smaller than that of the chessboard
    chessboard_points_cumulative_3d = np.delete(chessboard_points_cumulative_3d, 0, 1)
    roi_2D = get_boundary_corners_2D(chessboard_points_cumulative_3d)

    print("Calibration completed... \nPlace the box in the field of view of the devices...")

    """
            2: Measurement and display
            Measure the dimension of the object using depth maps from multiple RealSense devices
            The information from Phase 1 will be used here

            """

    # Enable the emitter of the devices
    device_manager.enable_emitter(True)

    # Load the JSON settings file in order to enable High Accuracy preset for the realsense
    device_manager.load_settings_json("./HighResHighAccuracyPreset.json")

    # Get the extrinsics of the device to be used later
    extrinsics_devices = device_manager.get_depth_to_color_extrinsics(frames)

    # Get the calibration info as a dictionary to help with display of the measurements onto the color image instead of infra red image
    calibration_info_devices = defaultdict(list)
    for calibration_info in (transformation_devices, intrinsics_devices, extrinsics_devices):
        for key, value in calibration_info.items():
            calibration_info_devices[key].append(value)

    k = 0
    # Continue acquisition until terminated with Ctrl+C by the user
    while 1:
         # Get the frames from all the devices
        frames_devices = device_manager.poll_frames()

        # Calculate the pointcloud using the depth frames from all the devices
        point_cloud = calculate_cumulative_pointcloud(frames_devices, calibration_info_devices, roi_2D)

        # Get the bounding box for the pointcloud in image coordinates of the color imager
        bounding_box_points_color_image, length, width, height = calculate_boundingbox_points(point_cloud, calibration_info_devices )

        k+=1

        print("k,roi_2D,point_cloud,length, width, height...",k,roi_2D,point_cloud,length, width, height)

        # Draw the bounding box points on the color image and visualise the results
        visualise_measurements(frames_devices, bounding_box_points_color_image, length, width, height)

        # for (device_info, frame) in frames_devices.items():
        #   device = device_info[0]  # serial number
        #   color_image = np.asarray(frame[rs.stream.color].get_data())
        #   # Visualise the results
        #   cv2.imshow('Color image from RealSense Device Nr: ' + device, color_image)
        #   cv2.waitKey(1)

except KeyboardInterrupt:
    print("The program was interupted by the user. Closing the program...")

finally:
    device_manager.disable_streams()
    cv2.destroyAllWindows()

if name == "main": run_demo()

![Uploading 1111.png…]()

point_cloud = calculate_cumulative_pointcloud(frames_devices, calibration_info_devices, roi_2D)，The point cloud information for this line of code is empty，How should I debug to find out the cause？

[MartyG-RealSense]

Hi @kay2019-art What distance from the checkerboard is the camera placed at, please? The ideal 'sweet spot' distance of the box_dimensioner_multicam project is 1 meter. Placing the camera nearer to or further from the board than 1 m can cause problems, such as a bounding box that does not fit around the observed box on the checkerboard.

May I also confirm whether the object placed on your checkerboard is a box or an irregularly shaped object?

[kay2019-art]

If the distance is one meter or more, there are both regular and irregular objects, but the results are measured as point clouds, and the data is empty. May I ask if using the checkerboard as the negative map is the two-dimensional coordinate point information of the regular image? Can the irregular objects be measured?

[MartyG-RealSense]

The box_dimensioner_multicam project is not designed to display pointcloud data. Instead, it generates a pointcloud for each attached camera placed around the checkerboard (it works with one camera despite the multicam name of the project) and then uses them to calculate a bounding box and overlay it over the object on the RGB image along with the volumetric measurements of the object.

The purpose of the checkerboard is to calibrate together the positions of multiple cameras relative to each other, which is why the object does not need to be placed on the board until this initial calibration has been completed. It is possible to adapt the project to remove the need for the checkerboard to be present if only one camera is being used.

---

It is rare to attempt to use this project for irregular objects instead of square boxes, as it would be difficult to accurately fit a single square bounding box around such an object.

There was a recent case where it was suggested that multiple different shaped bounding boxes could be used on irregular objects to better map around their contours.

https://support.intelrealsense.com/hc/en-us/community/posts/30888032025107-Dimensional-Analysis-of-an-object-using-realsense-D435

In that case, I suggested that a possible approach for creating a bounding box for irregularly shaped objects would be to use OpenCV contour code to form a bounding box around the object from rectangles and circles.

https://docs.opencv.org/3.4/da/d0c/tutorial_bounding_rects_circles.html

[kay2019-art]

@MartyG-RealSense I used a single camera, but it didn't show the length, width and height, and I couldn't get the point cloud data. If you use a single camera, can you detect irregular images without using the checkerboard as the base map?

[kay2019-art]

[MartyG-RealSense]

https://github.com/IntelRealSense/librealsense/issues/9951 and https://github.com/IntelRealSense/librealsense/issues/10054 discuss a RealSense user's attempts to adapt the box_dimensioner_multicam code to work without the chessboard by storing and using the intrinsics of the single camera instead.

box_dimensioner_multicam is almost always used with regular-shaped objects such as boxes though, so there is not enough information available for me to offer a prediction on the likelihood of success with irregular objects, unfortunately.

Are you positioning the camera at the corner of the chessboard facing inwards towards the object, and not directly overhead?

[kay2019-art]

I put the camera on the top of the checkerboard, when the checkerboard was detected, put the box on the checkerboard, there has been no reaction, you can not show the effect, the point cloud information is empty, what is the reason

[kay2019-art]

I want to debug it to see if the code can be executed on the terminal with sudo, but the device cannot be found when executed in pycharm. How should I operate

[MartyG-RealSense]

Are you using the 6x9" chessboard image provided by the box_dimensioner_multicam project?

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/box_dimensioner_multicam/pattern_chessboard.png

A RealSense user at https://github.com/IntelRealSense/librealsense/issues/11763 who was running the project in Pycharm had problems with the program that were resolved once the chessboard image at the link above was used, as they had been using their own chessboard image.

[MartyG-RealSense]

Hi @kay2019-art Do you have an update about this case that you can provide, please? Thanks!