Camera intrinsics when predicting/training on other datasets

[alexnwang]

Awesome work! Finally an unsupervised depth+motion model that works out of the box!

I'm looking to inference and/or train on BDD100K which doesn't have camera calibration details. In particular, I'm interested in the predicted flow projected into 2D space.

• If I'm inferencing with the Waymo or Nuscenes trained checkpoints, how impactful would it be to use approximate BDD intrinsics.
• What would be the correct approach to finetune a Waymo or Nuscenes checkpoint without ground truth camera calibration?

Thanks!

[YihongSun]

Thank you for your interest in our work!

Dynamo-Depth does make the assumption of GT camera intrinsics during training. For inference, depth, odometry, and motion segmentation predictions do not depend on the camera intrinsics but the flow predictions do. Please refer to issue #2 for more details.

• For inferencing with intrinsics from another dataset, since the independent motion predictions are made directly in the pseudo-3D space, different intrinsics may impact the flow predictions, but we did not look into the magnitude of this effect.
• For training without GT camera calibration, a potential solution would be to jointly estimate camera intrinsics during training as shown in this work. Due to the unsupervised nature of our work, we didn't investigate the optimal strategy for finetuning on another dataset. But I would speculate two potential solutions: 1) If there is a substantial domain gap, it may be better to ignore the checkpoints and train directly in the target domain with all proposed stages. 2) If there is a reasonable similarity in domain (e.g., similar contexts, ego-motion, and moving agents), extending the final finetuning stage with the released checkpoints may work better.

Feel free to reply if you have any other questions or concerns!