search

NVlabs / nvdiffrec

Official code for the CVPR 2022 (oral) paper "Extracting Triangular 3D Models, Materials, and Lighting From Images".
Other
2.09k stars 222 forks source link

Created Colab for those with nb without gpus. Enjoy ;D #30

Open neurall opened 2 years ago

neurall commented 2 years ago

https://github.com/neurall/nvdiffrecColab/

olemolvig commented 2 years ago

thanks! what kind of input (and where would it go)? Looking to test on small sets of 1-10 images to generate textured mesh.

neurall commented 2 years ago

I created this just yesterday and was able to run just run included demo dataset yet. As for how to create custom dataset for to reconstruct from your own images=create nerf/nerd dataset. My guess asking authors by creating a new issue "how to train on own images?" will yeld much faster and better answer. but since that part of docs is missing there and I hope will be added soon by them, For now I guess. following https://github.com/sxyu/svox2 how to do custom nerf dataset using colmap could work too, it worked for me back then and datasets are claimed as nerf compatible. I will try this today and update repo with what I found. But the authors seem to require special alpha pass . so you can't even use insta ngp datasets ? https://github.com/NVlabs/nvdiffrec/issues/26#issuecomment-1120138250

jmunkberg commented 2 years ago

Thanks for setting up the Colab example, @neurall !

10 images are on the lower side I think. We typically use 50+views in our tests.

Correct, we need foreground segmentation masks. In the synthetic NeRF datasets, that is already stored in the alpha channel of the images. Apart from adding mask, you can mostly follow https://github.com/bmild/nerf#generating-poses-for-your-own-scenes , or just modify any of the dataset readers in https://github.com/NVlabs/nvdiffrec/tree/main/dataset to suit your data.

olemolvig commented 2 years ago

Great, I will follow progress here.

@jmunkberg I am an historian of science and technology at Vanderbilt working on using Digitial reconstruction technologies (like photogrammetry, etc) to rebuild 3d artifacts to then interact with in VR. This approach seems like it will eventually be a more robust version of photogrammetry, at least giving plausible results with less input data. I would love to learn more about the process and pipeline going forward if you would be up for connecting irl.

kpjmcg commented 2 years ago

Thanks for this! I am also interested in utilizing this for personal purposes. I am an archaeologist seeking to use this for surface reconstruction and topological study.

neurall commented 2 years ago

Just an Useful hint for those searching for how to do your own datasets. This seems to be very fruitful thread. https://github.com/NVlabs/nvdiffrec/issues/3