Any suggestions for outdoor single object reconstruction?

[liuxinhai]

Example of my data（polycam）：

My training script: ns-train monosdf --pipeline.model.sdf-field.use-grid-feature True --pipeline.model.sdf-field.use-appearance-embedding True --pipeline.model.sdf-field.geometric-init True --pipeline.model.sdf-field.inside-outside False --pipeline.model.sdf-field.bias 0.8 --pipeline.model.sdf-field.beta-init 0.1 --pipeline.model.sdf-field.use-appearance-embedding True --pipeline.datamanager.train-num-images-to-sample-from 1 --pipeline.datamanager.train-num-times-to-repeat-images 0 --trainer.steps-per-eval-image 5000 --pipeline.model.background-model grid --vis wandb --experiment-name monosdf-bench --pipeline.model.mono-depth-loss-mult 0.1 --pipeline.model.mono-normal-loss-mult 0.05 --pipeline.model.eikonal-loss-mult 0.1 --pipeline.datamanager.train-num-rays-per-batch 2048 sdfstudio-data --data data/7-25-53_sdf/ --include_mono_prior True

Access to my data: https://drive.google.com/drive/folders/1nh_AQsbkdOHTPLOqT5NeTS0sKcrj43uk?usp=sharing

My results:

[niujinshuchong]

Did you already extract the mesh from your trained model?

[toomy0toons]

From what I understand the artifacts "white clouds" are nerf network trying to learn vast difference resulting from lights and reflective surfaces from outdoor, in the wild images. So you could try capturing under less light, or maybe less glossy object.

1. Data Collection May be could try capturing the car indoor with controlled lighting perhaps in garage?

2. Pose Estimation colmap does not work great for wild images, especially ones with lot of reflective surfaces. May be try colmap for harsher conditions suggested here. It does take longer but i found i it to produce poses better. https://github.com/autonomousvision/sdfstudio/issues/144#issuecomment-1611798435

3. Network network wise you could try models like neusW that takes appearence difference or lighting difference from object. You could also combine transient embedding from neusW to monosdf.

4. Loss I found multi view photo consistency loss from geo-neus very helpful in reducing floaters. You could turn on multi view photo consistency loss in monosdf model i think its defined as patch-warp.

would love to see your results good luck.

[liuxinhai]

Did you already extract the mesh from your trained model? This is my result.

[lexvandersluijs]

Hi,

Thanks for posting this, the fact that you got a result with Polycam data and SDF Studio encouraged me to push through.. I think I've finally resolved all code dependencies and such. I just have one question: the extract_monocular_priors script requires square images; how did you get these based on a Polycam dataset? Did you manually massage the data (images and metadata) or is there a script that does this?

Edit: never mind, I found that if you call process_nerfstudio_to_sdfstudio.py with the "--mono-prior" argument, that script will make the square images, correct the intrinsics, and call the Omnidata scripts to create the monocular cues. Hope I will get a proper mesh this time.

[ViokingTung]

Hi,

Thanks for posting this, the fact that you got a result with Polycam data and SDF Studio encouraged me to push through.. I think I've finally resolved all code dependencies and such. I just have one question: the extract_monocular_priors script requires square images; how did you get these based on a Polycam dataset? Did you manually massage the data (images and metadata) or is there a script that does this?

Edit: never mind, I found that if you call process_nerfstudio_to_sdfstudio.py with the "--mono-prior" argument, that script will make the square images, correct the intrinsics, and call the Omnidata scripts to create the monocular cues. Hope I will get a proper mesh this time.

this is my mesh trained with neus-facto, but there are some problems about wheels, I guess it's due to the lack of pictures from the perspective of the car chassis. I plan add this perspective and retrain. and the mesh is not smooth cause I add some parameters about ref to defeated reflection