search

bennyguo / instant-nsr-pl

Neural Surface reconstruction based on Instant-NGP. Efficient and customizable boilerplate for your research projects. Train NeuS in 10min!
MIT License
857 stars 84 forks source link

NeuRIS extension #3

Open pablovela5620 opened 2 years ago

pablovela5620 commented 2 years ago

Thanks so much for this project! Any chance to add an implementation of NeuRIS? https://jiepengwang.github.io/NeuRIS/ https://arxiv.org/pdf/2206.13597.pdf

The basic premise is that one incorporates a surface normal prior to generating higher fidelity meshes. The currently available implementation https://github.com/jiepengwang/NeuRIS takes many hours to train (around 9 on an A6000) so using the optimization you've added here would be amazing.

I would love to know your thoughts!

bennyguo commented 2 years ago

Thanks!

Adding normal/depth priors would be a very general-purpose enhancement and I'll definitely consider this. Also, testing on indoor scenes (supporting NeuRIS dataset format) is also in the plan. However, these may not happen very soon, so welcome to implement NeuRIS on our framework which is very easy to customize and extend: simply define a new model in models/ and a new training pipeline in systems/! (I'll add more illustrations on this in README)

pablovela5620 commented 2 years ago

Great, I'll take a look once you've added some more documentation as to how to implement new models/pipelines! Thanks again

pablovela5620 commented 2 years ago

Done some digging and hoping to contribute. I saw that you're currently bringing in some more dataset options.

I was hoping to add in geometric cues following NeuRIS or MonoSDF (two recent works that focus on the use of normal/depth cues)

I mostly wanted to make sure I wasn't bringing in a dataset that you're already in the process of doing. My plan is

  1. Implement Dataset that includes geometric cues (DTU/Scannet/Replica) based on MonoSDF repo
  2. Implement Normal map rendering from the volumetric renderer (I believe nerfacc already does depth?)
  3. Add relevant loss functions
  4. Implement patch match cost from NeuRIS

Let me know if this makes sense and any suggestions you may have. Thanks!

bennyguo commented 2 years ago

Thanks @pablovela5620! Very happy to have our first contributor!

About your plans, they all seem reasonable to me and are quite compatible with the current learning procedure:

  1. I'm currently working on a general COLMAP dataset, but I think it'll be fine to bring new datasets, since they're only new files in the datasets/ folder with little need to modify the training procedure. To implement new datasets, just (1) calculate self.directions based on intrinsic information; (2) read in camera poses self.all_c2w (in OpenGL coordinates); (3) read in images self.all_images and masks self.all_fg_masks (optional). Of course in your case you also have to read in the normal and depth maps, lets say self.all_normal_maps and self.all_depth_maps.
  2. This can be easily done with the newest version of nerfacc, which gives new render_weight_from_density/alpha functions (under my advice btw 😁).
  3. We probably need a new NeuRISSystem instead of using the original NeuSSystem. So just implement the relevant loss functions here.
  4. This is also loss-related, but requires patch-based sampling? If so, this also goes to the new NeuRISSystem since the ray sampling is currently implemented in the preprocess_data function.

Let me know if the above comments look good to you or if you have other questions. Thanks!