search

kwea123 / nerf_pl

NeRF (Neural Radiance Fields) and NeRF in the Wild using pytorch-lightning
https://www.youtube.com/playlist?list=PLDV2CyUo4q-K02pNEyDr7DYpTQuka3mbV
MIT License
2.74k stars 483 forks source link

Colmap fails on forward driving scene #59

Closed phongnhhn92 closed 3 years ago

phongnhhn92 commented 3 years ago

Hi, I am trying to train NERF on a sequence of a forward driving scene like this. ezgif com-optimize I have used the colmap script from LLFF to estimate the pose but it doesn't work for me. Need to run COLMAP Features extracted Features matched Sparse map created Finished running COLMAP, see /home/phong/VKITTI2/colmap_output.txt for logs Post-colmap Cameras 5 Images # 2 ERROR: the correct camera poses for current points cannot be accessed Done with imgs2poses here is the link to download the image. I wonder if NERF can not handle these sequences or is this from COLMAP ?

kwea123 commented 3 years ago

It should be a problem from COLMAP (which you can check by using its GUI), I think it's better to ask https://github.com/colmap/colmap/issues

Btw, I have worked on many real car driving scenes like this, including left turn or right turn, NeRF has no problem handling these, provided COLMAP correctly reconstructs the poses.

kwea123 commented 3 years ago

Is there a reason why you don't use the real KITTI? I think maybe COLMAP works better on real images.

phongnhhn92 commented 3 years ago

@kwea123 I try VKITTI first to see if NERF can work on the synthetic driving dataset. However, you said that you can train NERF with KITTI images? If it is then do you use COLMAP to estimate the poses (using LLFF code) or do you use the camera poses from the KITTI dataset and do something similar to the Blender dataset?

kwea123 commented 3 years ago

No, I tried with own dataset, not KITTI. I used COLMAP with default automatic reconstruction without any problem. Maybe you should try using bright scenes with many feature points (like road paintings) and reduce the sequence length (~50 consecutive frames) first.

kwea123 commented 3 years ago

LLFF code does nothing more than reading the COLMAP result and does some conversion, so if it fails it means COLMAP fails, it's COLMAP's problem.

zgojcic commented 3 years ago

Hi Kwea, just a quick question. When fitting the self-driving scenes did you use NDC space or not? Also, did you only use the front-facing cameras, or were multiple cameras (e.g. also side looking ones) used.

kwea123 commented 3 years ago

I use NDC space. Concerning the cameras, I use multiple forward-facing cameras, perhaps that makes the difference. Recently I found that monocular sequence doesn't work well in this kind of driving scenes.

zgojcic commented 3 years ago

This is interesting, I have observed that this works well for monocular sequence (but the extrinsic and intrinsic are taken over from the dataset and not estimates using COLMAP). Adding the side cameras causes weird artefacts on the images. I thought that this could come due to the NDC space because this then kind of becomes an inside out scene. However, NDC assumptions should also be violated in the case of left and right turns that you mentioned above (and this works for you apparently)...