search

NVlabs / instant-ngp

Instant neural graphics primitives: lightning fast NeRF and more
https://nvlabs.github.io/instant-ngp
Other
15.86k stars 1.91k forks source link

A quick question about the fox dataset #56

Closed xTayEx closed 2 years ago

xTayEx commented 2 years ago

Hello, thanks for your great paper. I would like to know how you generate the fox dataset. I take some photos of real world objects, and use the colmap2nerf.py script to process them. However. I didn't get a good training result using the new dataset. One of the original photos and the training result are shown bellow.

xTayEx commented 2 years ago

0001

image

mmalex commented 2 years ago

hi! the fox was just a few seconds iphone video, with a few frames selected to minimise blur and to cover a range of angles. you can see the chosen frames in the data/nerf/fox/images folder. the colmap2nerf script was used with default parameters apart from --aabb_scale 4 which basically says 'theres a small amount of background visible in the distance, outside the main bounding box'. see the nerf tips document for more info. the reconstruction you are showing above is indeed quite bad! how many images do you have? are they from a range of angles? have you checked if COLMAP was able to create accurate camera poses? (you can do that by using the COLMAP gui and running colmap manually). without seeing the dataset itself it's hard to know what's going wrong.

clutching at straws, perhaps the lack of contrast between the foreground and background is leading to the problem. The NeRF algorithm basically works by adjusting the density (shape) so that random training rays 'agree' on the color of the pixel. I can imagine that if the foreground and background objects are very similar in color, only the shadows will help to drive the algorithm towards convergence. so perhaps that could be it... but as I say, Id have to look at the data more to understand it. I would expect a scene of that complexity to be fittable fairly easily.

xTayEx commented 2 years ago

Thank you so much for your reply! I have made some progress in creating dataset. Instead of using the fan object shown above, now I am using a rubber duck to create my new dataset. Instant-ngp can reconstruct my duck, but there is so much 'dust' around it as shown below. image

I have checked my rubber duck dataset with COLMAP, the camera poses seems right. However, I found that the duck cant be reconstruct well in COLMAP. Would it be the cause of the 'dust'?

Below is the COLMAP analysis result and one of the rubber duck photos.

image

0052

xTayEx commented 2 years ago

What's more, to generate the dataset, I used colmap2nerf with default parameters apart from --aabb_scale 4.

mhalber commented 2 years ago

One tip that I have from my experiments - visualize cameras and ensure that the near distance is set appropriately. If I understand correctly, if your near clipping distance is too far from the actual focal plane, then the algorithm will never sample there, and hence it will never be able to clear out the "dust"

xTayEx commented 2 years ago

I use colmap2nerd.py in commit bb00a1 to regenerate my dataset, then it works magically! Someone who has the same problem can try it.

MiZhenxing commented 2 years ago

I use colmap2nerd.py in commit bb00a1 to regenerate my dataset, then it works magically! Someone who has the same problem can try it.

Great!

ttsesm commented 2 years ago

I am facing a similar problem, I have a set of images then using the colmap2nerf with the default values I extracted the camera poses: image but then the reproduced mesh looks really bad (used res 512): image if I zoom in I can see the object in the center, though in really bad quality but all these noise around is really bad: image while the input images are looking similar to the following: DSC00160 any idea how I could improve this?

Thanks.

ZepSbosnia commented 2 years ago

Hi, I've faced the same problem. I think that the 'dust' is caused by the the picture blur. Indeed I've preprocessed the images removing the background and the results really improved: before pidgeot_colmap_instant after pidgeot_colmap_instant_nobk the input was like : 000 an running rembg p path/to/the/images path/to/images/processed becomes : download

the question is how to use the code for more complicated scenes...

zhangsirya commented 1 year ago

Hi, I've faced the same problem. I think that the 'dust' is caused by the the picture blur. Indeed I've preprocessed the images removing the background and the results really improved:

Hello, may I ask how you remove the background? After removing the background, won't the black background be recreated?

ZepSbosnia commented 1 year ago

Hi, it depends how you remove the background. In my case i have set it transparent using RGBA color space (alpha channel is the transparent one) then saving in png format instant nerf won't consider the transparent colors. while to get better resolution of the obj i suggest to decrease the aabb value

zhangsirya commented 1 year ago

Wow, thank you for your response! I will try it out right away.