The evaluation results do not achieve the effect in the paper

[ArSpi]

I train models in the MatrixCity/aerial dataset with RTX 3090 24G, but the evaluation results are not good.

{
 "ours_30000": {
  "SSIM": 0.3807186186313629,
  "PSNR": 14.57436752319336,
  "LPIPS": 0.6211459040641785
 },
 "cells": {},
 "cfg_args": {},
 "input.ply": {},
 "point_cloud": {},
 "cameras.json": {},
 "costs.json": {}
}

{
 "Average FPS": 36.160666823899135,
 "Min FPS": 21.782481797314,
 "Average Memory(M)": 8202.54135103007,
 "Max Memory(M)": 9335.87646484375,
 "Number of Gaussians": 14247061
}

The results are so bad that I doubt if I do something wrong. What I did are as follows.

1. download colmap data by Baidu Netdisk: https://pan.baidu.com/s/1zX34zftxj07dCM1x5bzmbA?pwd=1t6r and download MatrixCity/aerial image data by Baidu Netdisk: https://pan.baidu.com/s/187P0e5p1hz9t5mgdJXjL1g?pwd=hqnn#list/path=%2F
2. my ./data folder is like

   ├── data
   │   ├── images
   │   │   ├── aerial
   │   │   │   ├── train
   │   │   │   │   ├──block_all
   │   │   │   │   │   ├── sparse
   │   │   │   │   │   │   ├── 0
   │   │   │   │   │   │   │   ├── cameras.bin
   │   │   │   │   │   │   │   ├── points3D.bin
   │   │   │   │   │   │   │   ├── images.bin
   │   │   │   │   │   ├── input
   │   │   │   │   │   │   ├── 0000.png
   │   │   │   │   │   │   ├── 0001.png
   │   │   │   │   │   │   ├── ......
   │   │   │   │   │   │   ├── 5620.png
   │   │   │   ├── test
   │   │   │   │   ├──block_all_test
   │   │   │   │   │   ├── sparse
   │   │   │   │   │   │   ├── 0
   │   │   │   │   │   │   │   ├── cameras.bin
   │   │   │   │   │   │   │   ├── points3D.bin
   │   │   │   │   │   │   │   ├── images.bin
   │   │   │   │   │   ├── input
   │   │   │   │   │   │   ├── 0000.png
   │   │   │   │   │   │   ├── 0001.png
   │   │   │   │   │   │   ├── ......
   │   │   │   │   │   │   ├── 0740.png

3. edit run_citygs.sh and the edited part of run_citygs.sh is as follow

   
   TEST_PATH="data/images/aerial/test/block_all_test/"

COARSE_CONFIG="mc_aerial_coarse" CONFIG="mc_aerial_c36"

out_name="val" max_block_id=35 # 35 is 6 * 6 - 1 port=4041

4. run `bash scripts/run_citygs.sh`
5. There are some evaluation results below (results/00000.png-00004.png)
![00000](https://github.com/user-attachments/assets/9c3d40d4-7eca-4016-868c-67c7c5541b62)
![00001](https://github.com/user-attachments/assets/46a94fed-78a6-430b-9f47-280a8c9524eb)
![00002](https://github.com/user-attachments/assets/5eae7d07-1f05-4cf8-9f8a-1bb57569b20b)
![00003](https://github.com/user-attachments/assets/1aa8344b-d9cb-4fdf-8201-8e5688d03b7d)
![00004](https://github.com/user-attachments/assets/34e803e8-872e-4c9f-8e78-cdb9364f6c52)
What wrong with the operations? What should I do?

[ArSpi]

and there are the evaluation results of coarse model

"ours_30000": {
  "SSIM": 0.47955480217933655,
  "PSNR": 15.612959861755371,
  "LPIPS": 0.6106020212173462
 }

[nurbanu170399]

How did you manage to run on RTX 3090 24G GPU? Mine kept giving out of memory error. Did you change any settings or anything? @ArSpi

[ArSpi]

How did you manage to run on RTX 3090 24G GPU? Mine kept giving out of memory error. Did you change any settings or anything? @ArSpi

I did not change any setting about the memory usage, and I just followed the tips in README.md step by step. I run the whole training pipeline without LOD, but encounter out-of-memory error when pruning the model.

[DekuLiuTesla]

and there are the evaluation results of coarse model

"ours_30000": {
  "SSIM": 0.47955480217933655,
  "PSNR": 15.612959861755371,
  "LPIPS": 0.6106020212173462
 }

Hi, @ArSpi , seems there is already something wrong in the coarse model. Under our settings the PSNR of coarse model can arrive at around 24.7. Besides, the result image order is different with mine. Please ensure that the images in renders folder align with gt folder. The out_name should be the folder containing ours_30000 folder. The following image shows my data structure. And in this case, it should be block_all_test. Please check if these tips can help.

[DekuLiuTesla]

How did you manage to run on RTX 3090 24G GPU? Mine kept giving out of memory error. Did you change any settings or anything? @ArSpi

@nurbanu170399 Here is some tips: https://github.com/DekuLiuTesla/CityGaussian/issues/9#issuecomment-2290538506. Hope it to be useful.

[WangYu0611]

In fact, I think 24G video memory cannot perform this task, because I found that sometimes the GPU video memory usage is as high as 25G+, and your result is not the optimized effect

[DekuLiuTesla]

For MatrixCity, I'm afraid it is difficult to arrive the optimized result in our paper with 24G VRAM currently since this scene contains so many details and training data. An alternative is to increase the gradient threshold and compromise performance for lower memory cost. Besides, there is space left for optimization. Not all points of coarse model are required in training. We would optimize this part in coming CityGS V2.

[WangYu0611]

对于 MatrixCity，目前恐怕很难在 24G VRAM 下达到我们论文中的优化结果，因为这个场景包含太多细节和训练数据。另一种方法是增加梯度阈值并牺牲性能以降低内存成本。此外，还有优化的空间。训练中不需要所有粗模型点。我们将在即将推出的 CityGS V2 中优化这部分。

cityGS V2 , cool

[ArSpi]

and there are the evaluation results of coarse model

"ours_30000": {
  "SSIM": 0.47955480217933655,
  "PSNR": 15.612959861755371,
  "LPIPS": 0.6106020212173462
 }

Hi, @ArSpi , seems there is already something wrong in the coarse model. Under our settings the PSNR of coarse model can arrive at around 24.7. Besides, the result image order is different with mine. Please ensure that the images in renders folder align with gt folder. The out_name should be the folder containing ours_30000 folder. The following image shows my data structure. And in this case, it should be block_all_test. Please check if these tips can help.

Thank you! Sorry that I deal with other things these two weeks. Now I just install colmap and process the images to generate cameras.bin, images.bin and points3D.bin. As a result, I get the excellent metrics. { "ours_30000": { "SSIM": 0.8651838302612305, "PSNR": 27.37071990966797, "LPIPS": 0.20437228679656982 } } Thank you for your advices!

[mistletoe235]

Could you please share the .ply file you obtained after training?

[ArSpi]

Could you please share the .ply file you obtained after training?

Please check https://pan.baidu.com/s/1U7KMV5Py1nQbgQ2nJOuFvA?pwd=jt94. I got it by using colmap to process images, not training. I am not sure whether it is what you want or not.