No points merged

[nurbanu170399]

I did the training on my own dataset. When it came to merge.py, no points can be found to merge except block number 10. I checked point_cloud.ply files of every cell and it looks like there are no points saved in the .ply files except for cell 10. I dont know why. How could i check if points are being saved to .ply file? The training took almost 2 days for 315 images. Why could this possibly happen? GPU 0 is available. 36 Merged 0 points from block 0 from iteration 30000. Merged 0 points from block 1 from iteration 30000. Merged 0 points from block 2 from iteration 30000. Merged 0 points from block 3 from iteration 30000. Merged 0 points from block 4 from iteration 30000. Merged 0 points from block 5 from iteration 30000. Merged 0 points from block 6 from iteration 30000. Merged 0 points from block 7 from iteration 30000. Merged 0 points from block 8 from iteration 30000. Merged 0 points from block 9 from iteration 30000. Merged 3929483 points from block 10 from iteration 30000. Merged 0 points from block 11 from iteration 30000. Merged 0 points from block 12 from iteration 30000. Merged 0 points from block 13 from iteration 30000. Merged 0 points from block 14 from iteration 30000. Merged 0 points from block 15 from iteration 30000. Merged 0 points from block 16 from iteration 30000. Merged 0 points from block 17 from iteration 30000. Merged 0 points from block 18 from iteration 30000. Merged 0 points from block 19 from iteration 30000. Merged 0 points from block 20 from iteration 30000. Merged 0 points from block 21 from iteration 30000. Merged 0 points from block 22 from iteration 30000. Merged 0 points from block 23 from iteration 30000. Merged 0 points from block 24 from iteration 30000. Merged 0 points from block 25 from iteration 30000. Merged 0 points from block 26 from iteration 30000. Merged 0 points from block 27 from iteration 30000. Merged 0 points from block 28 from iteration 30000. Merged 0 points from block 29 from iteration 30000. Merged 0 points from block 30 from iteration 30000. Merged 0 points from block 31 from iteration 30000. Merged 0 points from block 32 from iteration 30000. Merged 0 points from block 33 from iteration 30000. Merged 0 points from block 34 from iteration 30000. Merged 0 points from block 35 from iteration 30000. Saving merged 3929483 point cloud to output/stadium/point_cloud/iteration_30000/point_cloud.ply Done GPU 0 is available. Loading trained model at iteration 30000 [23/08 11:16:55] Reading camera 1/2 [23/08 11:16:55] Reading camera 2/2 [23/08 11:16:55] [23/08 11:16:55] Train cameras: 2, Test cameras: 0 [23/08 11:16:55] Converting point3d.bin to .ply, will happen only the first time you open the scene. [23/08 11:16:55] Rendering progress: 0%| | 0/2 [00:00<?, ?it/s][ INFO ] Encountered quite large input images (>1.6K pixels width), rescaling to 1.6K. If this is not desired, please explicitly specify '--resolution/-r' as 1 [23/08 11:16:58] Rendering progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00, 1.35it/s] Average FPS: 40.3448 [23/08 11:17:00] Min FPS: 26.7174 [23/08 11:17:00] Average Memory: 2471.2698 M [23/08 11:17:00] Max Memory: 2552.1191 M [23/08 11:17:00] Number of Gaussians: 3929483 [23/08 11:17:00] Skip both train and test, render all views [23/08 11:17:00] GPU 0 is available.

Scene: output/stadium Method: ours_30000 Metric evaluation progress: 0%| | 0/2 [00:00<?, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /home/pc_5053/.cache/torch/hub/checkpoints/vgg16-397923af.pth 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 528M/528M [02:06<00:00, 4.36MB/s] Downloading: "https://raw.githubusercontent.com/richzhang/PerceptualSimilarity/master/lpips/weights/v0.1/vgg.pth" to /home/pc_5053/.cache/torch/hub/checkpoints/vgg.pth███████▉| 528M/528M [02:06<00:00, 4.35MB/s] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.12k/7.12k [00:00<00:00, 7.54MB/s] Metric evaluation progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [02:09<00:00, 64.73s/it] SSIM : 0.1954435 PSNR : 10.1279850 LPIPS: 0.6798524

And the PSNR is pretty bad too. But during training, i saw PSNR values around 25-26. What is wrong? Thank you in advance.

[DekuLiuTesla]

Hi, seems nothing is stored after training different blocks. Please check if the .ply file exists. If so, open it and check if there is anything in the file. If nothing is saved, then try use small save iteration and debug to find what's happening. The training time is related to the number of GPUs. If there is only one card the training can be extremely long with 36 blocks.

[nurbanu170399]

Hi, seems nothing is stored after training different blocks. Please check if the .ply file exists. If so, open it and check if there is anything in the file. If nothing is saved, then try use small save iteration and debug to find what's happening. The training time is related to the number of GPUs. If there is only one card the training can be extremely long with 36 blocks.

I feel like there is something wrong with data partitioning. When i print the first 5 rows of the points of each block, the first 4 rows are always the same.

[DekuLiuTesla]

@nurbanu170399 We recently retested the latest repository and seems no bug exists in the source code. Perhaps you can first try saving at iteration 1 by setting save_iterations to test if it works.

[nurbanu170399]

@nurbanu170399 We recently retested the latest repository and seems no bug exists in the source code. Perhaps you can first try saving at iteration 1 by setting save_iterations to test if it works.

I did. Its the same. No points are saved in point_cloud.ply file. There is only the header. In this case, i have another question. Even though there are no points saved to the point_cloud.ply file in output folder, i am still able to render the scene. If there is no points saved to point_cloud.ply files in cell folders, how am i able to render it?

[DekuLiuTesla]

Perhaps the rendering visits another .ply file. Please check what file is loaded.

[DekuLiuTesla]

@nurbanu170399 issue #25 seems to meet similar problems. Please check if it is the case.