Finding 3x4 projection matrix for a camera in CARLA

[ahrnbom]

I'm running CARLA 9.11 on Ubuntu 18.04.

I want to find 3x4 projection matrices for cameras in CARLA. In other words, I want to find a matrix P such that if I have a 3D point provided by something like vehicle.get_location() (the location part of CARLA's transforms) on the shape X =

x
y
z
1

, then I want to be able to get the corresponding 2D position in that camera by x2D ~ P * X, where x2D are the 2D position on format

x
y
1

.

I think this is a pretty common camera matrix format, but I cannot figure out how to get it from CARLA. I can get the location and rotation of the camera (as x, y, z, pitch, roll, yaw), but I have not been able to convert it to a projection matrix P.

The projection matrix P is typically composed as follows: P = K * [R t] where K is a 3x3 intrinsic matrix given by

[f 0 Cx
 0 f Cy
 0 0 1]

which I have found (as there are several examples computing it), R is a 3x3 rotation matrix and t is a 3x1 translation.

I have been using the formula from https://github.com/carla-simulator/carla/issues/2516 to get the rotation matrix from pitch, roll, yaw. To get the translation, I've just computed it as t = -R C where C is the camera center position (which I have).

However, doing so does simply not work. I seem to be missing something. Anybody got a clue what's wrong?

[ahrnbom]

Solved it! Turn out CARLA has inconsistent coordinate systems. Instead of P = K [R t], you instead do P = K F * [R t] where F = np.array([[ 0, 1, 0 ], [ 0, 0, -1 ], [ 1, 0, 0 ]], dtype=np.float32)

[GauleeJX]

please tell me how can you get K, particularly the f of K matrix. I don't know how to get the f of rgb camera in carla. appreciating

[ahrnbom]

So first you set the field of view (fov) and image size to whatever you want, like so:

fov = 90
im_size_x = 1280
im_size_y = 720
cam_bp = world.get_blueprint_library().find('sensor.camera.rgb')
cam_bp.set_attribute('image_size_x', str(im_size_x))
cam_bp.set_attribute('image_size_y', str(im_size_y))
cam_bp.set_attribute('fov', str(fov))

Then you can manually compute f, Cx and Cy like so:

f  = im_size_x /(2.0 * tan(fov * pi / 360))
Cx = im_size_x / 2.0
Cy = im_size_y / 2.0

Finally, you can build K by

K = np.array([[f, 0, Cx], [0, f, Cy], [0, 0, 1]], dtype=np.float64)

[Aligadoudou]

Could you please tell me how to get the RT value? I don't fully understand, thanks so much .

[ahrnbom]

As I mentioned earlier, you get t from t = -R @ C. To get R, see https://github.com/carla-simulator/carla/issues/2516

If you need to see actual code examples, you can take a look at https://github.com/ahrnbom/utocs inside cameras.py