search

yifanlu0227 / ChatSim

[CVPR2024 Highlight] Editable Scene Simulation for Autonomous Driving via LLM-Agent Collaboration
https://yifanlu0227.github.io/ChatSim
308 stars 20 forks source link

About multi-camera alignment #4

Closed szhang963 closed 3 months ago

szhang963 commented 7 months ago

Hi, I am very interested in your promising work. I have some questions about multi-camera alignment.

  1. Why did you use Metashape to refine the camera pose? Is the result better than COLMAP for multi-camera pose estimating?
  2. Did you only refine multi-camera extrinsic (cami2ego)? Then, Did you use the pose of ego car from the vehicle sensor to translate the camera coord to the world？

Thanks in advance.

yifanlu0227 commented 7 months ago

Hi!

  1. We follow READ: Large-Scale Neural Scene Rendering for Autonomous Driving to use MetaShape for camera poses. We didn't use COLMAP to estimate the position before, so we are not sure which of them is more effective. However, considering that Metashape is a commercial software, it may provide a more accurate pose. In any case, using the original pose from the waymo dataset will lead to poor rendering results, and re-calibrating is important for NeRF trainining.

  2. Yes, as written in the paper, the coordinate system was eventually transformed back to the world coordinate system.

yifanlu0227 commented 7 months ago

There are many existing ways to use COLMAP for estimating camera poses. We also consider introducing relevant scripts to provide another open source and free choice.

szhang963 commented 7 months ago

Thanks for your active reply.

szhang963 commented 7 months ago

Hi, I have a question about metashape. Have you fixed each camera's intrinsic to estimate camera poses?

I followed the script to reconstruct Waymo multi-camera scenes, but the intrinsic of the camera sensor was modified and only one sensor existed. Could you provide some workflow or settings for multi-camera reconstruction, such as setting multi-camera intrinsic from the Waymo dataset as default parameters for metashape?

Thank you for your reply.

yifanlu0227 commented 7 months ago

We use 3 front-view cameras (FRONT, FRONT-LEFT, FRONT-RIGHT) in our ChatSim.

In Metashape, we assume they share the same intrinsics. We adopted the estimated camera intrinsics instead of using the original waymo intrinsics.

Screenshot 2024-03-05 at 11 39 29

The code will be released in the next day or two, and you can follow our calibration and conversion process of multi-camera alignment.

szhang963 commented 7 months ago

Thanks. I am eagerly looking forward to your code release.