search

dcharatan / flowmap

Code for "FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descent" by Cameron Smith*, David Charatan*, Ayush Tewari, and Vincent Sitzmann
https://cameronosmith.github.io/flowmap/
MIT License
872 stars 84 forks source link

About results #30

Closed wuqun-tju closed 4 months ago

wuqun-tju commented 4 months ago

I run the overfit.py using 15 frames of co3d_bench, but THE 3D results is wrong. Can you explain it ?

image

dcharatan commented 4 months ago

It's hard to tell what's happening with just the point cloud. Can you provide some more details on the scene, the configuration you ran, etc.?

In general, there are two failure modes for FlowMap:

  1. Rarely, FlowMap will fall into a "hollow face minimum" where depths are inverted, kind of like this.
  2. If RAFT optical flow estimation gives bad results, then the FlowMap results will correspondingly be bad.

If you connect FlowMap to wandb, you can see the predicted (FlowMap) and "ground-truth" (RAFT) optical flows alongside the predicted depth maps. This will hopefully provide some insight into what's happening.