Incorrect Bounding Box Position, Rotation, and Size with Actors Having Roll and Pitch in CARLA 9.14

[aiglesias-vicom]

Hello,

I am using CARLA version 9.14 on Ubuntu, and I am working on labeling falling obstacles in my simulation. To achieve this, I recorded obstacles falling onto the road and developed a custom labeling tool to annotate the objects. This approach has worked perfectly until I began introducing actors with both pitch and roll.

In the following image, you can see a falling static.prop.box01 and its corresponding bounding box. There is a clear discrepancy in the position, rotation, and size of the bounding box when compared to the actual object.

I am labeling the actors' bounding boxes using the world coordinate system and also storing the point cloud data in the same system. The following code snippet is used to retrieve the bounding box of an actor:

# Actor bbox
obj_bbox = obj.bounding_box
# Get bbox size
bbox_size = [obj_bbox.extent.x, obj_bbox.extent.y, obj_bbox.extent.z]

# BBox coords in the object coordinate system
cords = create_bb_points(bbox_size)
bb_transform = carla.Transform(obj_bbox.location)
bb_obj_matrix = get_matrix(bb_transform)

# Object transform in world coordinate system
obj_transform = obj.get_transform()
obj_world_matrix = get_matrix(obj_transform)
bb_world_matrix = np.dot(obj_world_matrix, bb_obj_matrix)

# Vertices in the (left handed) world coordinate system
world_cords = np.dot(rh2lf_mat_4x4, np.dot(bb_world_matrix, np.transpose(cords)))

This code is based on the function vehicle_to_world from the repo. Additionally, I use other functions from the same repository to help with the annotation process:

def create_bb_points(vehicle):
    cords = np.zeros((8, 4))
    extent = vehicle.bounding_box.extent
    cords[0, :] = np.array([extent.x, extent.y, -extent.z, 1])
    cords[1, :] = np.array([-extent.x, extent.y, -extent.z, 1])
    cords[2, :] = np.array([-extent.x, -extent.y, -extent.z, 1])
    cords[3, :] = np.array([extent.x, -extent.y, -extent.z, 1])
    cords[4, :] = np.array([extent.x, extent.y, extent.z, 1])
    cords[5, :] = np.array([-extent.x, extent.y, extent.z, 1])
    cords[6, :] = np.array([-extent.x, -extent.y, extent.z, 1])
    cords[7, :] = np.array([extent.x, -extent.y, extent.z, 1])
    return cords

### Get transformation matrix from carla.Transform object
def get_matrix(transform):
    rotation = transform.rotation
    location = transform.location
    c_y = np.cos(np.radians(rotation.yaw))
    s_y = np.sin(np.radians(rotation.yaw))
    c_r = np.cos(np.radians(rotation.roll))
    s_r = np.sin(np.radians(rotation.roll))
    c_p = np.cos(np.radians(rotation.pitch))
    s_p = np.sin(np.radians(rotation.pitch))
    matrix = np.matrix(np.identity(4))
    matrix[0, 3] = location.x
    matrix[1, 3] = location.y
    matrix[2, 3] = location.z
    matrix[0, 0] = c_p * c_y
    matrix[0, 1] = c_y * s_p * s_r - s_y * c_r
    matrix[0, 2] = -c_y * s_p * c_r - s_y * s_r
    matrix[1, 0] = s_y * c_p
    matrix[1, 1] = s_y * s_p * s_r + c_y * c_r
    matrix[1, 2] = -s_y * s_p * c_r + c_y * s_r

    matrix[2, 0] = s_p
    matrix[2, 1] = -c_p * s_r
    matrix[2, 2] = c_p * c_r
    return matrix    

As mentioned, I had no issues with this method until I started testing scenes with objects that have roll and pitch. To help reproduce the issue, I have provided the CARLA recording:

falling_obstacle.log

I have also included the labeled data in OpenLABEL format. After weeks of testing and debugging, I suspect the error is related to how CARLA calculates bounding boxes, as well as the position and rotation of actors when roll and pitch are involved.

Any guidance or suggestions on resolving this issue would be greatly appreciated.

Thank you for your time and assistance.

[PatrickPromitzer]

HI, for me it looks like a calculation error I had before.

btw. bounding box size is extent 2 `bbox_size = [obj_bbox.extent.x 2, obj_bbox.extent.y 2, obj_bbox.extent.z 2]`

I wrote a code to get the center point of the object, and with that you may calculate the points you want. I used the Carla function to convert the the coordinate into world position.

bounding_box_size = [
    obj.bounding_box.extent.x * 2,
    obj.bounding_box.extent.y * 2,
    obj.bounding_box.extent.z * 2,
]
bounding_box_center = [
    obj.bounding_box.location.x,
    obj.bounding_box.location.y,
    obj.bounding_box.location.z,
]
bounding_box_rotation = [
    obj.bounding_box.rotation.roll,
    obj.bounding_box.rotation.pitch,
    obj.bounding_box.rotation.yaw,
]

actor_transform = obj.get_transform()
bounding_box_location = carla.Location(
    x=bounding_box_center[0],
    y=bounding_box_center[1],
    z=bounding_box_center[2]
)
actor_transform.transform(bounding_box_location)
actor_center = [
    bounding_box_location.x,
    bounding_box_location.y,
    bounding_box_location.z
]

If that doesn't work, you can try calculating the bounding box data of the object yourself. I made a script to do that, or you use the JSON file calculate_carla_actor_bounding_box_v2.zip

[aiglesias-vicom]

Hello @PatrickPromitzer,

First of all, thank you for your help. However, it seems that with the code you've provided, the results are still not correct, as can be seen in the image.

Perhaps it was my fault, and I didn’t use it correctly. In my first comment, I provided the recording (the map is Thown10HD). Could you please try to visualize the bounding box to determine if it was my mistake or an error in Carla?

[gsmario]

Hello! As you can see in this image, the bounding box of the asset box01 its correct. Let me know if I can help you with anything else.

[aiglesias-vicom]

Hello @gsmario,

You're correct that the bounding box appears accurate when checking the assets. However, the issue we're encountering seems to be related to execution time. Specifically, when Carla is running and pitch and roll are applied to an object, the bounding box returned via the Python API does not seem to be correct.

I've provided a short recording in Town10 that demonstrates the issue. It would be great if you could test it using this recording to help replicate the problem.