NerfStudio + SuGaR - Initital tests

[rebecca-lay3rs]

Bureau_lightroom_nerfstudio_format_hloc

• changed folder organization to have :

DATA FOLDER

├── dataset
│   ├── colmap (for NerfStudio)
│   │   ├── sparse
│   │   │   ├── 0
│   │   │   │   ├── camera.bin etc
│   ├── distorted (coming from SuGaR)
│   ├── images (for both)
│   ├── sparse (for Sugar)
│   ├── transforms.json (for Nerfstudio)

source_path='/root/nerfstudio_SuGaR/data/Bureau_lightroom_nerfstudio_format_hloc'

---

OUTPUT FOLDER

├── 2024-XX-XX_XXXXXX
│   ├── nerfstudio_models (for NerfStudio)
│   ├── point_cloud (coming from ns-export gaussian-splatting)
│   │   ├── iteration_7000
│   │   │   ├── point_cloud.ply
│   ├── cfg_args (for Sugar)
│   ├── cameras.json (for Sugar)
│   ├── input.ply (for Sugar)

• other nerfstudio files

• Sugar is gonna understand that this output folder is the output of the GS model when instead is the output of Splatfacto

model_path='/root/nerfstudio_SuGaR/outputs/Bureau_lightroom_nerfstudio_format_hloc/splatfacto/2024-02-21_121102'

---

cfg_args file -> contains simple info on the model

Namespace(sh_degree=3, source_path='/root/nerfstudio_SuGaR/data/Bureau_lightroom_nerfstudio_format_hloc', model_path='outputs_GS/Cloitre_Gal2_E2_Statues_proche_bis', images='images', resolution=-1, white_background=False, data_device='cuda', eval=False)

---

cameras.json -> needs to be converted from the Nerfstudio format to the Sugar format

• using the code in convert_json.py inside Sugar folder

---

• we call the Sugar train.py on the output of Splatfacto

python train.py -s /root/nerfstudio_SuGaR/data/Bureau_lightroom_nerfstudio_format_hloc/ -c /root/nerfstudio_SuGaR/outputs/Bureau_lightroom_nerfstudio_format_hloc/splatfacto/2024-02-21_121102/ -r "sdf" --low_poly True --poisson_depth 8 --experiment_name Bureau_lightroom_nerfstudio_format_hloc/

the code turns but we get the error:

WARNING: No gaussians available for sampling.                                               
coarse_sdf.py:708

• TODO:
• check conventions
• try to open Splatfacto on GS viewer

[rebecca-lay3rs]

Checking conventions

• using notebook of neuralangelo

• the transform_matrix are shifted but the scale is the same
• the change in scale is carried inside nerfstudio, not in the transforms.json

Bureau Sugar/GS

Bureau Splatfacto

1, 1, 1

[rebecca-lay3rs]

1, 1, -1

[rebecca-lay3rs]

1, -1, 1

[rebecca-lay3rs]

1, -1, -1

[rebecca-lay3rs]

-1, -1, -1

[rebecca-lay3rs]

-1, -1, 1

[rebecca-lay3rs]

Scaling Splatfacto pcd to GS/Sugar pcd convention:

• In http://51.159.156.62:8088/notebooks/SuGaR/converting_json.ipynb# , I use Open3D to read the .ply file as tensor (t.io) and rescale the .ply the Splatfacto (way smaller than the one from GS) to the bigger .ply to GS/Sugar

• Then I open the scaled .ply of Splatfacto in CloudCompare and I manually adjust the blue they are equal, colorfull they are different

after reloading the .ply with more info of header

tranformation_matrix [17:24:42] -0.844333350658 0.535432219505 -0.020336233079 10.036691665649 0.087618708611 0.100525870919 -0.991068899632 7.630033969879 -0.528605878353 -0.838574349880 -0.131791219115 -7.633952617645 0.000000000000 0.000000000000 0.000000000000 1.000000000000

second trial with all header of .ply [09:46:44] -0.839185059071 0.543805301189 0.006650578231 10.154912948608 0.062881372869 0.109169028699 -0.992032289505 7.957352161407 -0.540198445320 -0.832080483437 -0.125808238983 -7.050346851349 0.000000000000 0.000000000000 0.000000000000 1.000000000000

Running Sugar on Splatfacto outputs

Test 1: with cameras.json of Sugar

• called now cameras_original_sugar_alignement.json inside http://51.159.156.62:8088/tree/nerfstudio_SuGaR/outputs/Bureau_lightroom_nerfstudio_format_hloc/splatfacto/test_Bureau_splatfacto/

python train.py -s /root/nerfstudio_SuGaR/data/Bureau_lightroom_nerfstudio_format_hloc/ -c /root/nerfstudio_SuGaR/outputs/Bureau_lightroom_nerfstudio_format_hloc/splatfacto/test_Bureau_splatfacto/ -r "sdf" --low_poly True --poisson_depth 8 --experiment_name Bureau_lightroom_nerfstudio_format_hloc/

• points3D.ply missing on the sparse/0/ folder, train.py stoped after coarse mesh extraction
• running the rest of the code separately

python train_refined.py -s /root/nerfstudio_SuGaR/data/Bureau_lightroom_nerfstudio_format_hloc/ -c /root/nerfstudio_SuGaR/outputs/Bureau_lightroom_nerfstudio_format_hloc/splatfacto/test_Bureau_splatfacto/ -i 7000 -m output/coarse_mesh/Bureau_lightroom_nerfstudio_format_hloc/sugarmesh_3Dgs7000_sdfestim05_sdfnorm02_level03_decim200000.ply -o output/refined/Bureau_lightroom_nerfstudio_format_hloc/ -v 200000 --experiment_name Bureau_lightroom_nerfstudio_format_hloc/

python extract_refined_mesh_with_texture.py -s /root/nerfstudio_SuGaR/data/Bureau_lightroom_nerfstudio_format_hloc/ -c /root/nerfstudio_SuGaR/outputs/Bureau_lightroom_nerfstudio_format_hloc/splatfacto/test_Bureau_splatfacto/ -i 7000 -m output/refined/Bureau_lightroom_nerfstudio_format_hloc/sugarfine_3Dgs7000_sdfestim05_sdfnorm02_level03_decim200000_normalconsistency01_gaussperface1/15000.pt -o output/refined_mesh/Bureau_lightroom_nerfstudio_format_hloc --experiment_name Bureau_lightroom_nerfstudio_format_hloc/

Test 2: with transform_sugar_convention.json

• I take the Nerfstudio .json (obtained with Nerfstudio script and in Nerfstudio conventions) and transform to Sugar convention with the code in the neuralangelo notebook.
• Then I use the http://51.159.156.62:8088/notebooks/SuGaR/converting_json.ipynb# code to put this .json in the Sugar format
• I run the scripts separately to change the experiment_name

python train.py \ -s /root/nerfstudio_SuGaR/data/Bureau_lightroom_nerfstudio_format_hloc_Nerfjson_Sugarconv/ \ -c /root/nerfstudio_SuGaR/outputs/Bureau_lightroom_nerfstudio_format_hloc/splatfacto/test_Bureau_splatfacto/ \ -r "sdf" \ --low_poly True \ --poisson_depth 8 \ --experiment_name Bureau_lightroom_nerfstudio_format_hloc_Nerfjson_Sugarconv/

• to see if commands below are needed python extract_mesh.py \ -s /root/nerfstudio_SuGaR/data/Bureau_lightroom_nerfstudio_format_hloc/ \ -c /root/nerfstudio_SuGaR/outputs/Bureau_lightroom_nerfstudio_format_hloc/splatfacto/test_Bureau_splatfacto/ \ -r "sdf" \ --low_poly True \ --poisson_depth 8 \ --experiment_name Bureau_lightroom_nerfstudio_format_hloc_Nerfjson_Sugarconv/

-c outputs_GS/Bureau/ -i 7000 -m output/coarse/Bureau_sdf_low_poly/sugarcoarse_3Dgs7000_sdfestim02_sdfnorm02/15000.pt -l 0.3 -d 200000 -o output/coarse_mesh