Implementation of Neuralangelo

[denghilbert]

Hi authors,

I notice you implement Neuralangelo in sdfstudio. I am a little bit confused.

According to my understanding, the structure of Neuralangelo is based on Neus (or monoSDF, which borrows something from volsdf).

The key idea is that we feed "position", "view direction", "appearance embedding (or env latent code)", and maybe "hash feature" like instant-ngp into MLP to get SDF and combined feature. The combined feature, along with "positioin" etc., are then used for predicting RGB.

Neuralangelo has only two main differences (at least for me): 1) the use of numerical gradient and 2) coarse to fine training. I wonder if you have the comparison to verify those two addon? Besides, my test on monosdf (borrow the code from this repo) for an outdoor scene is disappointing. 1) Slow training 2) converge at bad (or I should say no) geometry, as shown in the following figures. I didn't use GT depth or normal for supervision but the results shouldn't be that bad.... If you have some idea about this or a special training trick?

Thank you so much!

Best

[MvWouden]

Perhaps this issue can offer more info/perspectives: https://github.com/autonomousvision/sdfstudio/issues/120

[denghilbert]

Perhaps this issue can offer more info/perspectives: #120

Thanks! But after 2 days training with 4 3090s, my geometry still seems to be extremely bad.

[niujinshuchong]

@denghilbert How is your data preprocessed? Did you enable background model?

[denghilbert]

@denghilbert How is your data preprocessed? Did you enable background model?

Data processing follows nerf++.

Oh, I see you implemented an extra bg network and merged it with the final fg rendering results. For my implementation, I didn't add that. Also, I didn't use coarse to fine strategy. I will continue to verify whether both factors are crucial for the IN THE WILD scene. Update results later.

[3a1b2c3]

No way with a 3600,can i lower any of the settings or reduce data?

[niujinshuchong]

@3a1b2c3 You can try to reduce the number of training rays and use a smaller model such as reducing log2_hashmap_size, hash_features_per_level, hidden_dim, and hidden_dim_color.

[3a1b2c3]

Thanks, lets try

[denghilbert]

BTW, I find the colmap setting in this documentation can not work well for "in the wild" images. I suggested using the following colmap setting if the user needs to calculate poses with varying illumination images:

    # Feature extraction
    feature_extractor_cmd = [
        f"{colmap_cmd} feature_extractor",
        f"--database_path {colmap_dir / 'database.db'}",
        f"--image_path {image_dir}",
        "--ImageReader.single_camera 0",
        f"--ImageReader.camera_model {camera_model.value}",
        #"--ImageReader.camera_model SIMPLE_RADIAL",
        "--SiftExtraction.max_image_size 1280",
        "--SiftExtraction.estimate_affine_shape 0",
        "--SiftExtraction.domain_size_pooling 1",
        "--SiftExtraction.use_gpu 1",
        "--SiftExtraction.max_num_features 16384",
        "--SiftExtraction.gpu_index 0"
    ]
    # Feature matching
    feature_matcher_cmd = [
        f"{colmap_cmd} {matching_method}_matcher",
        f"--database_path {colmap_dir / 'database.db'}",
        "--SiftMatching.guided_matching 1",
        "--SiftMatching.use_gpu 1",
        "--SiftMatching.max_num_matches 65536",
        "--SiftMatching.max_error 3",
        "SiftMatching.gpu_index 0"
    ]
    mapper_cmd = [
        f"{colmap_cmd} mapper",
        f"--database_path {colmap_dir / 'database.db'}",
        f"--image_path {image_dir}",
        f"--output_path {sparse_dir}",
        "--Mapper.tri_min_angle 3.0 ",
        "--Mapper.filter_min_tri_angle 3.0",
        "--Mapper.ba_global_use_pba 1"
    ]