Pulling Monocular Mode

[Gaozhongpai]

Hi,

Thanks for sharing the codes. I am trying to train monocular mode on pulling_soft_tissues data.

First, I use DepthAnythingto get the monodepthas:

Then, I train the model with command: python train.py -s dataset/endonerf_sample_datasets/pulling_soft_tissues --port 6017 --expname endonerf/pulling_mono --configs arguments/endonerf/pulling_mono.py

The PSNRcan reach 41.36. However, the point cloud generated is not good as:

Metric evaluation progress: 100%|██████████████████████████████████████| 8/8 [00:00<00:00, 10.29it/s] Scene: output/endonerf/pulling_mono/ SSIM : 0.9527130 Scene: output/endonerf/pulling_mono/ PSNR : 36.5221901 Scene: output/endonerf/pulling_mono/ LPIPS: 0.0783771 Scene: output/endonerf/pulling_mono/ RMSE: 7.6948177

Is there anything wrong with my setup? Thanks a lot.

[yifliu3]

Hi,

Thanks for your attention. The generated point cloud you display is the initialized points, not the generated point cloud after model training. To produce the generated point cloud, you can use render.py and change the reconstruct keyword to True. Surprisingly, using such initialization can also produce a reasonable point cloud based on our experiments.

The reason that the initialized point cloud seems not good is caused by the scale-unaware mono-depth estimation. The depth anything model would produce depth maps of different scales for different images, thus directly combining them as we used in the current version would cause point cloud misalignment.

To generate a better initialization, we have modified the code and only used the first image for the Gaussian initialization, achieving a better initialization effect:

Based on the modified initialization, the generated point cloud of the test set is shown here:

With updated metric for pulling: Scene: output/endonerf/pulling_mono SSIM : 0.9477461 Scene: output/endonerf/pulling_mono PSNR : 35.8486176 Scene: output/endonerf/pulling_mono LPIPS: 0.0843981 Scene: output/endonerf/pulling_mono RMSE: 5.0240870

Thanks a lot for your observation!

[Gaozhongpai]

Based on the modified initialization, the generated point cloud of the test set is shown here:

With updated metric for pulling: Scene: output/endonerf/pulling_mono SSIM : 0.9477461 Scene: output/endonerf/pulling_mono PSNR : 35.8486176 Scene: output/endonerf/pulling_mono LPIPS: 0.0843981 Scene: output/endonerf/pulling_mono RMSE: 5.0240870

Thanks a lot for your observation!

Scene: output/endonerf/pulling_mono/ SSIM : 0.9562252 Scene: output/endonerf/pulling_mono/ PSNR : 37.1470413 Scene: output/endonerf/pulling_mono/ LPIPS: 0.0617576 Scene: output/endonerf/pulling_mono/ RMSE: 11.9361257

Thank you for your update. I got higher numbers using your updated code.