Carla bounding box

[mzahran001]

I am trying to extract 3D bounding box and 2D bounding box using camera sensor or the lidar sensor. I am using Caral Version 0.9.3 It is obvious how to extract the bounding box using the old api from the previous issues . But I am wondering how to return the bounding box for each object in the scene after the huge modification in carla API.

[mzahran001]

I tried this code in the automatic_control.py file to visulize the 3D bounding box

in the while True:

for vehicle in world.world.get_actors().filter('vehicle.*'):
    print(vehicle.bounding_box)
    # draw Box
    transform = vehicle.get_transform()
    bounding_box = vehicle.bounding_box
    bounding_box.location += transform.location
    world.world.debug.draw_box(bounding_box, transform.rotation)

But I found that there is huge lag in the box relative to the car

Is there any suggestions how to solve this problem ?

[mzahran001]

I found that this problem do not appear with static cars it only appears with moving ones. This another example from ROS with the same problem.

I tried to play with life_time=-1.0, persistent_lines=True in the function draw_box but unfortunately it did not help much. Still the bounding box is not in synch with the actual car

[nsubiron]

I would say the problem is that there is a lag of at least one frame from the moment you read the car position until you draw the box.

So things happen in this order

• Simulator sends "world tick" with the current transform of every vehicle
• In Python you calculate bounding box based on this transform and do a draw request
• Simulator parses this request

But another tick may have happened just before parsing the request, thus the box will be drawn on the next frame. At this point there is no way of guaranteeing that. Once #1258 is fixed, you might have the chance of accomplish this, but still there is a very small window between send world and parse requests. Ideally this should be done with synchronous mode #1244.

On top of that there is another lag that can be affecting, if you send draw requests for each vehicle, it may happen that the simulator cannot parse all the request in a single frame (it has a limited amount of time each frame allocated for this). Thus you'll get a growing queue of requests adding some lag. This hopefully will be fixed with #784.

Good news is, that those three issues are planned to be implemented in the coming release by the end of February :grin:

[mzahran001]

Thank You @nsubiron for your detailed response. I have another question, what is the reference point for the box that returns from carla (Carla.World, Vehicle, our the sensor)? and what is the units of the produced box is it in centimeters or meters?

The question in another way. If I have boundingbox.location and boundingbox.extent, how can I draw the bounding box on the lidar bird view for example?

[Soolek]

We have been dealing a lot with this lately. Good results started to appear once used custom drawing of 3d boxes client-side (a lot of time went into dealing with tranformations):

It is still not perfect, you can see the the green box shifting a bit when driving on a curb. This gets worse with speed, despite using:

• synchronous mode
• setting sensor_tick to 0.1 (10fps) https://carla.readthedocs.io/en/latest/cameras_and_sensors/#sensorcamerargb
• running server and client on the same machine Funny enough, our bounding boxes are now getting a bit ahead of cars seen on the screen.

What would really help is to have synchronous sensor listen function as it gets really bad the more sensors you set. We tried to synchronize it but sometimes (with multiple 4K cameras and lidar) the listen function would continue to receive data for more than 30 seconds after the last world.tick() was sent.

Also, the _world.debug.drawbox seems to have some delay on it's own... :)

[nsubiron]

I see... actually I made a mistake in my previous comment. Even with #1258 fixed, all the commands you send have effect in the next frame. I think that's why there is always a delay in the debug box. Plus on top of that there is no batch command for debug messages, so you can have an extra lag by sending each box individually, and on top of that I'm not sure how debug draws are implemented in UE4 nor if they have any extra delay.

Drawing the boxes in the client-side is probably the best approach.

Also, if you are able to share the code please do, a lot of people in the community would be very thankful :)

What would really help is to have synchronous sensor listen function as it gets really bad the more sensors you set.

Rendering is done in a separate thread in Unreal Engine, and this thread can be up to two frames behind the main thread. After that, the buffer is passed to another thread that sends it to the client. This is good for performance because while we are rendering and sending the image the main thread is already calculating the next frame, we minimize the time any thread is idle, but of course with big images you can have a significant delay.

But that's the point of the synchronous mode, you can block the main thread until you receive the images. In the synchronous_mode.py example that we release, we use a Python queue to wait for the image, so we call client.tick() only after receiving the image. That worked for the example. But yeah, this is a pretty new feature, so please report if you find something not working as expected.

[Manojbhat09]

@Soolek This is a very interesting implementation, great work!. Hope that this becomes a python tool in carla to be used in client side.

[Soolek]

Oh, I wasn't aware of the synchronomous_mode.py - it contains a very crucial synchronisation example that we have been struggling with (i.e. how to match video frame with transformation data): world.wait_for_tick().frame_count should be matched with what ends in sensor.listen() property .frame_number! Thanks to this I will be able to synchronise buffers (bounding boxes on image), thank you!

Once we polish the client bounding boxes I promise to rewrite the code to manual_control.py as we are using a sandbox opencv-based client at the moment

[Soolek]

Also, if you are able to share the code please do, a lot of people in the community would be very thankful :)

I have just created a pull request: https://github.com/carla-simulator/carla/pull/1383

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[germanros1987]

The functionality was merged! Thanks @Soolek