Need advice on custom data with poor result

[HungNgoCT]

Hi, Thank you for amazing work. I have used instant-ngp, nerfstudio to generate mesh. However, sdfstudio impress me by mesh generation. When I use sample data, I always get good render and mesh. However, when I use custom data, results were always poor although I have done experiments and follow all discussions in this page.

Could you give me advice to process with this data: https://drive.google.com/file/d/1hLnwXon29bc4SiyuxiV32VAfQnZY2gTd/view?usp=share_link As screen shot of above shared data like this:

The steps I have done as followings:

• Used colmap to generate transforms.json file as usual in instant-ngp
• Tested with instant-ngp, give a very good result (video 1 below)
• Tested with nerfstudio by: ns-train neus-facto --pipeline.model.sdf-field.inside-outside False --vis viewer instant-ngp-data --data D:\Hung\sdfstudio\data_cu\elephantNERFSDF\PNG_colmap Give a not good render
• I also use scripts ns-process-data, then render with nerfstudio, give a good result
• used script process_nerfstudio_to_sdfstudio.py to convert from nerfstudio json file to meta_data.json file, and run with sdfstudio (neus-facto), got a very poor result (video 2).

I have checked very careful discussion in #2 (https://github.com/autonomousvision/sdfstudio/issues/2), and #49 (https://github.com/autonomousvision/sdfstudio/issues/49), but rendering result is very poor like #49.

Below videos are rendering with instant-ngp Video 1: https://user-images.githubusercontent.com/110378166/222356933-3112bdb8-3c96-4c30-a516-fc3af3e88706.mp4

And rendering with sdfstudio video 2: https://user-images.githubusercontent.com/110378166/222357045-c8175d1c-f59c-49db-82ab-1f9c5b81f6fa.mp4

The mesh I got from sdfstudio like this:

Highly appreciate with any help.

[niujinshuchong]

Hi, I don't have access permission to your data, could you update your link?

[HungNgoCT]

Hi, I don't have access permission to your data, could you update your link?

Thank you in advance @niujinshuchong , I have corrected the link. Please help download again. Note that if I use ns-process-data and process_nerfstudio_to_sdfstudio.py, cannot see anything at rendering.

[niujinshuchong]

Hi, the main object in your scene is not centered after nerfstudio-data parser. You could try to add the code here https://github.com/autonomousvision/sdfstudio/blob/master/nerfstudio/data/dataparsers/nerfstudio_dataparser.py#L196 to roughly center the object

# center the objects
poses[:, 0, 3] += 1.0        

And you could run with following command

ns-train volsdf --pipeline.model.sdf-field.use-grid-feature False --pipeline.model.sdf-field.hidden-dim 256 --pipeline.model.sdf-field.num-layers 2 --pipeline.model.sdf-field.num-layers-color 2 --pipeline.model.sdf-field.use-appearance-embedding False --pipeline.model.sdf-field.geometric-init True --pipeline.model.sdf-field.inside-outside False  --pipeline.model.sdf-field.bias 0.5 --pipeline.model.sdf-field.beta-init 0.1 --pipeline.datamanager.train-num-rays-per-batch 2048 --pipeline.model.background-model mlp --pipeline.model.near-plane 0.1 --pipeline.model.far-plane 2.0 --pipeline.model.overwrite-near-far-plane True --vis wandb --experiment-name volsdf-dtu106 nerfstudio-data --data  data/PNG_colmap/

I run it for 10k iterations and got something like this

Maybe you could try to tune the network architecture/batch size/image resolutions to get better results.

[HungNgoCT]

Hi, the main object in your scene is not centered after nerfstudio-data parser. You could try to add the code here https://github.com/autonomousvision/sdfstudio/blob/master/nerfstudio/data/dataparsers/nerfstudio_dataparser.py#L196 to roughly center the object

# center the objects
poses[:, 0, 3] += 1.0        

And you could run with following command

ns-train volsdf --pipeline.model.sdf-field.use-grid-feature False --pipeline.model.sdf-field.hidden-dim 256 --pipeline.model.sdf-field.num-layers 2 --pipeline.model.sdf-field.num-layers-color 2 --pipeline.model.sdf-field.use-appearance-embedding False --pipeline.model.sdf-field.geometric-init True --pipeline.model.sdf-field.inside-outside False  --pipeline.model.sdf-field.bias 0.5 --pipeline.model.sdf-field.beta-init 0.1 --pipeline.datamanager.train-num-rays-per-batch 2048 --pipeline.model.background-model mlp --pipeline.model.near-plane 0.1 --pipeline.model.far-plane 2.0 --pipeline.model.overwrite-near-far-plane True --vis wandb --experiment-name volsdf-dtu106 nerfstudio-data --data  data/PNG_colmap/

I run it for 10k iterations and got something like this

Maybe you could try to tune the network architecture/batch size/image resolutions to get better results.

So wonderful! Thank you for your help, @niujinshuchong I will try to follow your guidance and update my result.

[suptimq]

Hi, the main object in your scene is not centered after nerfstudio-data parser. You could try to add the code here https://github.com/autonomousvision/sdfstudio/blob/master/nerfstudio/data/dataparsers/nerfstudio_dataparser.py#L196 to roughly center the object

# center the objects
poses[:, 0, 3] += 1.0        

And you could run with following command

ns-train volsdf --pipeline.model.sdf-field.use-grid-feature False --pipeline.model.sdf-field.hidden-dim 256 --pipeline.model.sdf-field.num-layers 2 --pipeline.model.sdf-field.num-layers-color 2 --pipeline.model.sdf-field.use-appearance-embedding False --pipeline.model.sdf-field.geometric-init True --pipeline.model.sdf-field.inside-outside False  --pipeline.model.sdf-field.bias 0.5 --pipeline.model.sdf-field.beta-init 0.1 --pipeline.datamanager.train-num-rays-per-batch 2048 --pipeline.model.background-model mlp --pipeline.model.near-plane 0.1 --pipeline.model.far-plane 2.0 --pipeline.model.overwrite-near-far-plane True --vis wandb --experiment-name volsdf-dtu106 nerfstudio-data --data  data/PNG_colmap/

I run it for 10k iterations and got something like this

Maybe you could try to tune the network architecture/batch size/image resolutions to get better results.

I wondered how you determined if the scene is centered or not.

[f-dy]

This would fix the centering issue ("focus" method) https://github.com/nerfstudio-project/nerfstudio/pull/1543

[niujinshuchong]

@suptimq I manually checked the camera poses in the viewer with --vis viewer. Thanks @f-dy for the suggestion. It would be great to have an automatic pose centering method.

[f-dy]

I just submitted a PR.

[a1445605672]

Is this what the code looks like?