extrinsics overlay details

[Goulustis]

Thanks for the dataset!

I'm also trying to project rgb to events or the other way around. I see that extrinsic overlay is provided to align the cameras. However, I'm very confused as to how to apply it to the given extrinsics of each camera to align the cameras together. Any information will be of great help (eg. are they camera to vicon?).

Thanks in advance!

[aftersomemath]

Hi!

If I remember correctly they are camera to rig (e.g. where the vicon thinks the center of the camera rig is).

We provide a tool that can reproject the RGB image into an event camera's frame (up to occlusions and field of view overlap of course) using the depth information and intrinsics. See here

Hope that helps!

[Goulustis]

Thanks!

I have found the math I'm looking for in the linked file. However the notation is quite confusing. I was wondering if you could give insight to what the T_* symbols from the two places below mean? (for example, how to interpret the subscript, c2o and gr, in T_c2o and T_gr)

here

T_gr = load_extrinsics(bgr_file)   
T_gc = load_extrinsics(file)  
T_cr = apply_transform(inv_transform(T_gc), T_gr)

and here

T_c2o = interpolate_pose(bgr_frame_info['ts'], all_poses[key])  
T_c2c1 = apply_transform(T_c2o, inv_transform(T_c1o))  
T_r2c1 = apply_transform(inv_transform(T_cr), T_c2c1)

Thanks!

[aftersomemath]

It should be: T_gr is rgb camera to camera rig. T_gc is event camera to camera rig. T_cr is rgb camera to event camera. T_c2o is object to event camera frame at time 2 (in this case bgr_frame_info['ts']) T_c2c1 is event camera frame at time 1 to event camera frame at time 2. T_r2c1 is event camera frame at time 1 to rgb camera frame at time 2.

This last transform (plus the depth information in the event camera frame) allows an RGB image at one time to be reprojected into the event camera frame at another time. This makes it possible to render high fps RGB data in the event camera frame even though the RGB camera ran at lower rate.

I am not sure of your familiarity with coordinate frame conventions, but if you are wondering why the subscripts are written in the order they are you can check out the "subscript cancellation rule" explained on page 91 of Modern Robotics (pdf page 111).

[Goulustis]

I have solved my problem. Thanks for the help!