fuse RGB and Lidar at identical timestamp but failed

[bittuh]

Hi LGSVL team,

I'm trying to fuse RGB image and Lidar output from cyber_recorder, with 60/30 FPS respectively, in order to get RGBD-like data (i.e. colorize point cloud according to camera image). However the same (moving) objects in RGB and Lidar appears differently even the timestamp is identical (in ms). I had ensured timestamp coming from simulator by setting MODE_MOCK as doc suggested.

Is there any way to get the correct RGB & Lidar mapping ITO time?

[EricBoiseLGSVL]

Is the issue in the speed of bridge transmission? Any ideas @kamkolak-lge ?

[kamkolak-lge]

There are multiple things that might be wrong here, so I'll ask a few questions to narrow it down:

• Does your lidar sensor have Cubemap Size parameter available? If not, you might be using a version that has timestamps offset by data processing time. This is most likely the case if the issue doesn't happen when vehicle is still and no moving objects are visible. Current version of lidar sensor will publish a precise timestamp of when data was captured, not including further possible processing time.
• Does the image appear to be uniformly offset in relation to point cloud, or does it vary in different positions of the image? If points towards the center match better, camera intrinsics may be wrong.
• Did you take into account the offset between lidar and camera positions?

[bittuh]

Thanks for your help @kamkolak-lge

• As you said, I didn't see Cubemap Size param in lidar configuration from LGSVL Web interface. How can I get access to the current version of lidar implementation? I'm using official 2021.1 simulator currently and knowing that I can rebuild simulator from source. Doesn't it lead to inconsistency between my simulator and the Web interface?
• The correctness of camera intrinsic and extrinsic between camera & lidar have been verified by good colorized point cloud of still object. Only color and 3D point of moving ones seems not matched.

[kamkolak-lge]

@bittuh You can clone the lidar repository and follow the asset build instructions to build the bundle and upload it through the web interface.

As for the compatibility - the change I mentioned was done after the 2021.1 tag, which means it might not be possible to build it using that particular version. It should be compatible with Simulator version 2021.2.

Edit: regarding inconsistency between simulator and web interface: all bundles include information about version of the simulator they were built in (more specifically, asset format version). When you build the bundle yourself, it should be compatible with the version of the simulator it was built in.

Edit 2: It seems that public lidar bundle for version 2021.2 already includes the change you need. It should be used when you move to Simulator version 2021.2 - you shouldn't have to build anything yourself. Cubemap Size parameter might not visible, but it seems to be a bug. Let me know if moving to version 2021.2 fixed the issue with moving objects.