NerfStudio + Gaussian Splatting - Initial Tests

rebecca-lay3rs commented 9 months ago

Installation check

checking if the NerfStudio installationis working well

Lego

ns-train nerfacto --data data/lego/ blender-data

Cloitre_Gal2_E2_Statues_proche_21_images

ns-process-data images --data data/Cloitre_Gal2_E2_Statues_proche_21_images/images --output-dir data/Cloitre_Gal2_E2_Statues_proche_21_images_nerfstudio_format --num-downscales 0

Bureau

nerfacto

ns-train nerfacto --data data/Bureau_lightroom_nerfstudio_format_hloc/ nerfstudio-data

splatfacto

code generated .ply for GS input
train time is slow 300 ms/iter

rebecca-lay3rs commented 9 months ago

results for splatfacto

rebecca-lay3rs commented 9 months ago

Official GS

rebecca-lay3rs commented 9 months ago

Angel - Splatfacto Tests

data: Cloitre_Gal1_C1_angel alignment: COLMAP of gaussian splatting

1. copying the data from GS to Nerfstudio

2. making a `colmap` folder

cp -r sparse/ colmap/
add points3D.ply to sparse/0

3. converting the COLMAP data of GS to .json of Nerfstudio using:

ns-process-data images --data data/Cloitre_Gal1_C1_angel/images --output-dir data/Cloitre_Gal1_C1_angel --num-downscales 0 --skip-colmap

4. Running Splatfacto

(2024-02-27_144622)

ns-train splatfacto --experiment-name Cloitre_Gal1_C1_angel nerfstudio-data --center-method none --data data/Cloitre_Gal1_C1_angel/

- from checkpoint: (2024-02-27_161104)

ns-train splatfacto --load-checkpoint outputs/Cloitre_Gal1_C1_angel/splatfacto/2024-02-27_144622/nerfstudio_models/step-000018000.ckpt --output-dir outputs/ --experiment-name Cloitre_Gal1_C1_angel nerfstudio-data --center-method none --data data/Cloitre_Gal1_C1_angel/

- exporting .ply file ns-export gaussian-splat --load-config outputs/Cloitre_Gal1_C1_angel/splatfacto/2024-02-27_161104/config.yml --output-dir outputs/Cloitre_Gal1_C1_angel/splatfacto/2024-02-27_161104/

5. organizing the Splatfacto Angel output folder

├── 2024-XX-XX_XXXXXX
│   ├── nerfstudio_models (for NerfStudio)
│   ├── point_cloud (coming from ns-export gaussian-splatting)
│   │   ├── iteration_7000
│   │   │   ├── point_cloud.ply
│   ├── cfg_args (for Sugar)
│   ├── cameras.json (for Sugar)
│   ├── input.ply (for Sugar)

6. convert the .ply output file of Splatfacto to Sugar/GS convention using Cloud Compare

multiple output_splatfacto.ply by 6.6 (.ply from Splatfacto too small compare to GS one)
align the output_splatfacto.ply and the output_GS.ply manually

[17:58:04] 0.997457563877 -0.008917156607 -0.070703245699 0.342103064060 -0.071006946266 -0.040277168155 -0.996662318707 0.069106496871 0.006039667875 0.999148726463 -0.040807943791 -0.280390352011 0.000000000000 0.000000000000 0.000000000000 1.000000000000

use the ICP registration in Cloud Compare to almost perfectly align both .ply
save the edited output_splatfacto.ply
remove with text editor all the header parts from "scaled_XXX" to XXXX

7. convert the transform.json of Splatfacto to Sugar/GS convention

http://51.159.156.62:8088/notebooks/SuGaR/converting_json.ipynb#
8. convert the transform.json of Splatfacto to Sugar/GS format
http://51.159.156.62:8088/notebooks/SuGaR/converting_json.ipynb#

9. Run train.py of Sugar

python train.py -s /root/nerfstudio_SuGaR/data/Cloitre_Gal1_C1_angel/ -c /root/nerfstudio_SuGaR/outputs/Cloitre_Gal1_C1_angel/splatfacto/2024-02-27_161104/ -r "sdf" --low_poly True --poisson_depth 8 --experiment_name Cloitre_Gal1_C1_angel/

rebecca-lay3rs commented 9 months ago

Other Montmajour datasets

to test if the splatfacto model is robust
to test if Splatfacto + Sugar gives similar results or not

Cloitre_Gal1_2_A1_broken_head

inference looks good but the camera poses are "too far" from object meaning I cannot turn the object neither get very close to it

Cloitre_Gal1-2_A2_sheep

same problem as above

Lay3rss / nerfstudio_SuGaR