search

NVlabs / instant-ngp

Instant neural graphics primitives: lightning fast NeRF and more
https://nvlabs.github.io/instant-ngp
Other
15.81k stars 1.9k forks source link

Tips for photos for good NeRFs #887

Open barney2074 opened 2 years ago

barney2074 commented 2 years ago

Based on some tips Jonathan Stevens kindly provided to me for taking photos to get good quality NeRFs, and from my own experience, I've put together these tips.

Feel free to add or disagree

  1. Image quality is more important than image quantity- i.e. delete any imperfect photos
  2. SHARP imagery (most important! NeRF likes to grab onto motion blur and make things fuzzy)
  3. The subject/scene needs to be still - moving trees, people, etc become smeared and ghost like
  4. Consistent exposure - this helps reduce floaters
  5. Flat lighting (i.e. overcast skies) is better than full sunlight and resulting shadow/exposure issues
  6. A proper camera where you can control shutter speed and aperture (and therefore minimise blurring and maximise depth-of-field) is better than a phone
  7. If you've got access to a proper camera- a 'fast' lens is better than a slow one for sharpness and depth-of-field
  8. I've got better results from taking a little time with still photos, rather than video

To be decided:

  1. I haven't worked out yet if HDR is better than non-HDR. I'm guessing HDR is better if it picks out detail in shadows, as long as the exposure is consistent
  2. For the fast lens thing- I don't know if an extreme focal length lens (I've got an 11mm lens which is great, but kind of distorted) or fish-eye type lenses are good or bad for NeRFs

NeRF Gallery: https://youtu.be/aMRx4WvKXpc

Andrew

gfiameni commented 1 year ago

Thank @barney2074 for your tips. When do you take pictures from different angles of the object you want to reconstruct, how do you get camera information, including x, y coordinates?

barney2074 commented 1 year ago

That's what Colmap does- it reconstructs the scene by identifying common points in photos and calculating the camera positions and parameters

Refer:

https://colmap.github.io/tutorial.html

Harper714 commented 1 year ago

@barney2074 Have you tested on outdoor/unbounded forward-facing images? I always get black ghosts on the far backgrounds.

barney2074 commented 1 year ago

Hi @Harper714

Yes, sometimes get that too- not exactly sure what causes it- but I tend to crop out the far distance-. Recent changes in I-NGP means it's easier to render once cropped

Harper714 commented 1 year ago

Hi @Harper714

Yes, sometimes get that too- not exactly sure what causes it- but I tend to crop out the far distance-. Recent changes in I-NGP means it's easier to render once cropped

Thanks for sharing. Could you kindly elaborate on what you mean by recent changes in ngp?

barney2074 commented 1 year ago

@Harper714 A change was made which retains crop settings in the snapshot

https://github.com/NVlabs/instant-ngp/commit/54aba7cfbeaf6a60f29469a9938485bebeba24c3 https://github.com/NVlabs/instant-ngp/issues/873#issuecomment-1296292395

Previously- the crop from interactive session was not saved in a snapshot- so a render would show everything

Harper714 commented 1 year ago

@barney2074

Many thanks. I will also try it.

jdiazram commented 1 year ago

Hi @barney2074, excellent tips. I have a question: What type, model, or brand of camera can you recommend to take good photos or videos to use them in NeRF for training?

barney2074 commented 1 year ago

HI @jdiazram

I don't think there is a specific camera- just good quality photos. I would have said a 'proper' camera with a decent lens (like a DSLR)- but to be honest, I've got pretty good results with both an iPhone (I've got an iPhone 12) or a GoPro (Hero 11) I'd say image quality is more important that ultra-high resolution- big images will slow down training, or you might run out of memory depending on GPU I'd also suggest looking at 'nerfstudio'- it's a bit easeir to use than instant-ngp, plus also works with LIDAR enabled IOS devices using Polycam app