Training Results on ScanNet

[AlbertHuyb]

Thanks for your nice work and released code! @jiepengwang

I follow your instructions to prepare the ScanNet dataset and train NeuRIS with the default configuration on scene0009_01.

However, after 160k iterations, I get the following render results.

And the following warning info continues appearing after the 50k iterations. The corresponding meshes file is also empty.

[Open3D WARNING] Read PLY failed: number of vertex <= 0.
[Open3D WARNING] Write PLY failed: mesh has 0 vertices.

I'm checking what's wrong and wondering how to get the correct result.

Do you have any suggestions? Thanks a lot!

[jiepengwang]

"I follow your instructions to prepare the ScanNet dataset and train NeuRIS with the default configuration on scene0009_01." Here, do you mean you don't use the GT camera poses and use them from openMVG? Do you check whether the inputs are correct? In general, after 10k iterations, there should be a coarse shape.

[AlbertHuyb]

Thanks for your reply!

I used the GT camera poses for ScanNet. I didn't run openMVG.

From my understanding of the preprocess steps, the preprocessing script generate cameras_sphere.npz from GT poses. And the cameras_sphere.npz is used in training.

I think I use the correct poses. How can I check the inputs to make sure they are correct?

Thanks for your kind help.

[AlbertHuyb]

If possible, could you please provide your input data under indoor/scene0009_01?

With your data as reference, perhaps I can find my problem.

Thank you!

[jiepengwang]

You can find the input data in the shared onedrive link in readme.md.

[AlbertHuyb]

@jiepengwang Thanks a lot for your help! Thank you for providing such a comprehensive open-source project. It's pretty cool and very convenient!

I re-run the experiment on scene_0009_01 with the downloaded data. And the render result seems correct this time.

However, the mesh seems incorrect neither. I visualize 00160000_reso128_scene0009_01.ply in meshlab. It seems like three layers of rooms are concatenated vertically.

And the comparison with groundtruth is shown in the following image. It seems that they are not in the same scale.

[AlbertHuyb]

And the reconstruction of scene_0085_00 also has similar problems.

00090000_reso128_scene0085_00.ply

[jiepengwang]

Hi, as illustrated in the paper, because of the SDF property, we can still get triangle faces at the invisible areas of the input images, when using marching cubes. You can remove the irrelevant faces by checking the visibility of the triangle faces to the input cameras. Or you can render depth maps then fuse them using tsdf fusion.

[AlbertHuyb]

Thanks for your reply! I understand the triangle faces at the invisible areas now.

And I noticed the validate_mesh mode in exp_runner.py script. I finally get high-quality mesh after running the following script:

python ./exp_runner.py --mode validate_mesh --conf ./confs/neuris.conf --gpu 0 --scene_name scene0009_01 --is_continue

The visualization of 00160000_reso512_scene0009_01_world.ply and comparison with groundtruth:

That's pretty good reconstructions.

[AlbertHuyb]

"I follow your instructions to prepare the ScanNet dataset and train NeuRIS with the default configuration on scene0009_01." Here, do you mean you don't use the GT camera poses and use them from openMVG? Do you check whether the inputs are correct? In general, after 10k iterations, there should be a coarse shape.

Hi @jiepengwang, sorry to bother you once more.

Regarding the camera poses of ScanNet, do you use the GT poses or openMVG poses?

I noticed that your data directory in OneDrive does not contain the point_cloud_scan.ply file.

As indicated by #L224 of scannet_data.py, this file in necessary if we use ScanNet data.

Because self.path_cloud_sfm is None here. The initialization arguments in #L229 of neuris_data.py does not contain path_cloud_sfm for ScanNet data preparation.

So I wonder how do you generate poses for ScanNet data in the OneDrive link.

Thanks!

[jiepengwang]

I use the GT poses to generate the training data. point_cloud_scan.ply is generated by fusing GT depth maps, which is not included in the shared data.

[AlbertHuyb]

I see, thank you!

And I find my problem in pre-processing.

I save the normal .npz files in the B-G-R channel order, instead of the correct R-G-B order.

For those who want to test NeuRIS on other ScanNet scenes, you can refer the following lines to generate .npz files from surface normal images.

def predict_normal(dir_neus, normal_method = 'snu'):
    # For scannet data, retraining of normal network is required to guarantee the test scenes are in the test set of normal network.
    # For your own data, the officially provided pretrained model of the normal network can be utilized directly.
    pred_save_dir = f'{dir_neus}/pred_normal'

    if normal_method == 'snu':
        # ICCV2021, https://github.com/baegwangbin/surface_normal_uncertainty
        logging.info('Predict normal')
        IOUtils.changeWorkingDir(dir_snu_code)
        os.system(f'python test.py --pretrained scannet --path_ckpt {path_snu_pth} --architecture BN --imgs_dir {dir_neus}/image/')

        if not os.path.exists(pred_save_dir):
            os.makedirs(pred_save_dir)

        for name in os.listdir(f'{dir_neus}/image/results/'):
            if name[:-4].endswith('pred_norm'):
                norm_data = (cv2.imread(os.path.join(f'{dir_neus}/image/results/',name))/255 - 0.5)*2
                np.savez(f"{pred_save_dir}/{name[:4]}.npz", arr_0=norm_data[...,::-1])
                cv2.imwrite(f"{pred_save_dir}/{name[:4]}.png", ((norm_data+1)*0.5*255).astype(np.uint8))

It would be better if you could include the normal .npz files generation into current pre-processing steps.

Thank you again for your patience and kind help @jiepengwang . And thanks for this pretty good open-source project.

I'll close this issue then.