Very bad training results

yuyunlong2002 commented 1 month ago

I produced very poor results when running Gaussian_stlatting, not only on my own dataset, but also on the MipNeRF360 scenes tandt_db provided by the training project. There is a significant gap between my training results and the results in the pre-trained model provided by the project, I don't know where the problem lies, I haven't changed any hyperparameters at all. The hyperparameter settings are the same as on the website：

        self.position_lr_init = 0.00016
        self.position_lr_final = 0.0000016
        self.position_lr_delay_mult = 0.01
        self.position_lr_max_steps = 30_000
        self.feature_lr = 0.0025
        self.opacity_lr = 0.05
        self.scaling_lr = 0.005
        self.rotation_lr = 0.001
        self.percent_dense = 0.01
        self.lambda_dssim = 0.2
        self.densification_interval = 100
        self.opacity_reset_interval = 3000
        self.densify_from_iter = 500
        self.densify_until_iter = 15_000
        self.densify_grad_threshold = 0.0002
        self.random_background = False
        super().__init__(parser, "Optimization Parameters")

I don't know if my hardware didn't meet the requirements, but I feel like the result shouldn't be so bad, my hardware is as follows:

GPU: Nvidia RTX2070super 8GB
RAM: 16 GB

My reconstruction results are compared with the pre trained results as follows： My Result: Pre-trained:

The observation results showed that there was a significant difference in the initial state of the splats, and as the splashes increased, *many large white spots appeared in the results. And I also found that there is a significant difference in memory size between point.cly in my iteration30000 and point.cly in pre trained iteration30000：

Not only is there an issue in this particular scenario, but all demo scenarios and my own datasets (train、trucks、playroom……）currently have the same problem, Can someone tell me where the problem lies?

o-o-cloud commented 1 month ago

I encountered the same problem during my experiment yesterday! wrong

yuyunlong2002 commented 1 month ago

@o-o-cloud hello，I was lucky enough to solve this problem after sending it out last night，I ran Gaussian Splatting again using Colab and found that the final result was good. I hope this can help you solve this problem. The problem is likely due to hardware issues with the computer, but I am still unsure why the hardware is affecting the training results

o-o-cloud commented 1 month ago

@yuyunlong2002 Thank you for your enthusiastic help, but I think my hardware is enough. RTX40 series I trained under ubuntu and visualized under windows

yuyunlong2002 commented 2 weeks ago

@kirikunkun Did you try it again on Colab I mentioned？I solved it by using colab，maybe my hardware is affecting the training results

@o-o-cloud hello，I was lucky enough to solve this problem after sending it out last night，I ran Gaussian Splatting again using Colab and found that the final result was good. I hope this can help you solve this problem. The problem is likely due to hardware issues with the computer, but I am still unsure why the hardware is affecting the training results

kirikunkun commented 2 weeks ago

@kirikunkun Did you try it again on Colab I mentioned？I solved it by using colab，maybe my hardware is affecting the training results

@o-o-cloud hello，I was lucky enough to solve this problem after sending it out last night，I ran Gaussian Splatting again using Colab and found that the final result was good. I hope this can help you solve this problem. The problem is likely due to hardware issues with the computer, but I am still unsure why the hardware is affecting the training results

Not yet, I'm trying to find a reason other than hardware, I used the 3080TI and it didn't turn out better than another worse machine. If in the end it is really the hardware reason, it will be the worst possible outcome

kirikunkun commented 2 weeks ago

@o-o-cloud I'm also having the same issue, have you solved it?

Thank you for your enthusiastic help, but I think my hardware is enough. RTX40 series I trained under ubuntu and visualized under windows

kirikunkun commented 1 week ago

I resolved the issue. I followed the instructions on Colab to clone from Git, which also cloned the folders in the submodules. Previously, I couldn’t clone two files in the submodules, so I downloaded them using links from the comments. Now, I suspect that the issue might have been caused by incompatible versions of the submodules. 微信图片_20241106154749

o-o-cloud commented 1 week ago

@kirikunkun I think you are right. I also downloaded them using links from the comments. So how can we download them in right ways？

o-o-cloud commented 1 week ago

I resolved the issue. I followed the instructions on Colab to clone from Git, which also cloned the folders in the submodules. Previously, I couldn’t clone two files in the submodules, so I downloaded them using links from the comments. Now, I suspect that the issue might have been caused by incompatible versions of the submodules.

我猜测我们是不是可以用中文交流

kirikunkun commented 1 week ago

我解决了这个问题。我按照 Colab 上的说明从 Git 克隆，Git 也克隆了子模块中的文件夹。以前，我无法在子模块中克隆两个文件，因此我使用评论中的链接下载了它们。现在，我怀疑该问题可能是由子模块的不兼容版本引起的。

我猜测我们是不是可以用中文交流

我直接运行了git clone --recursive https://github.com/camenduru/gaussian-splatting这一行他这次幸运的把两个子模块都下载下来了之前也会出现网络问题导致没下载下来子模块

o-o-cloud commented 1 week ago

感谢，那我将尝试在ubuntu上挂一个梯子，重新下载这两个子模块！

------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2024年11月6日(星期三) 下午4:05 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [graphdeco-inria/gaussian-splatting] Very bad training results (Issue #1013)

我解决了这个问题。我按照 Colab 上的说明从 Git 克隆，Git 也克隆了子模块中的文件夹。以前，我无法在子模块中克隆两个文件，因此我使用评论中的链接下载了它们。现在，我怀疑该问题可能是由子模块的不兼容版本引起的。

我猜测我们是不是可以用中文交流

我直接运行了git clone --recursive https://github.com/camenduru/gaussian-splatting这一行他这次幸运的把两个子模块都下载下来了之前也会出现网络问题导致没下载下来子模块

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

kirikunkun commented 1 week ago

感谢，那我将尝试在ubuntu上挂一个梯子，重新下载这两个子模块！ … ------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2024年11月6日(星期三) 下午4:05 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [graphdeco-inria/gaussian-splatting] Very bad training results (Issue #1013) 我解决了这个问题。我按照 Colab 上的说明从 Git 克隆，Git 也克隆了子模块中的文件夹。以前，我无法在子模块中克隆两个文件，因此我使用评论中的链接下载了它们。现在，我怀疑该问题可能是由子模块的不兼容版本引起的。我猜测我们是不是可以用中文交流我直接运行了git clone --recursive https://github.com/camenduru/gaussian-splatting这一行他这次幸运的把两个子模块都下载下来了之前也会出现网络问题导致没下载下来子模块 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

如果下载不行的话可以试试https://github.com/graphdeco-inria/diff-gaussian-rasterization和https://gitlab.inria.fr/bkerbl/simple-knn 这是我从我的git显示上复制下来的两个版本链接

o-o-cloud commented 1 week ago

很感谢，但是这两个链接我打不开，请问方便再发一个嘛！感谢感谢！！

------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2024年11月6日(星期三) 下午4:14 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [graphdeco-inria/gaussian-splatting] Very bad training results (Issue #1013)

感谢，那我将尝试在ubuntu上挂一个梯子，重新下载这两个子模块！ … ------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2024年11月6日(星期三) 下午4:05 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [graphdeco-inria/gaussian-splatting] Very bad training results (Issue #1013) 我解决了这个问题。我按照 Colab 上的说明从 Git 克隆，Git 也克隆了子模块中的文件夹。以前，我无法在子模块中克隆两个文件，因此我使用评论中的链接下载了它们。现在，我怀疑该问题可能是由子模块的不兼容版本引起的。我猜测我们是不是可以用中文交流我直接运行了git clone --recursive https://github.com/camenduru/gaussian-splatting这一行他这次幸运的把两个子模块都下载下来了之前也会出现网络问题导致没下载下来子模块 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

如果下载不行的话可以试试https://github.com/graphdeco-inria/diff-gaussian-rasterization和https://gitlab.inria.fr/bkerbl/simple-knn 这是我从git上复制下来的两个版本链接

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

kirikunkun commented 1 week ago

很感谢，但是这两个链接我打不开，请问方便再发一个嘛！感谢感谢！！ … ------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2024年11月6日(星期三) 下午4:14 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [graphdeco-inria/gaussian-splatting] Very bad training results (Issue #1013) 感谢，那我将尝试在ubuntu上挂一个梯子，重新下载这两个子模块！ … ------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2024年11月6日(星期三) 下午4:05 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [graphdeco-inria/gaussian-splatting] Very bad training results (Issue #1013) 我解决了这个问题。我按照 Colab 上的说明从 Git 克隆，Git 也克隆了子模块中的文件夹。以前，我无法在子模块中克隆两个文件，因此我使用评论中的链接下载了它们。现在，我怀疑该问题可能是由子模块的不兼容版本引起的。我猜测我们是不是可以用中文交流我直接运行了git clone --recursive https://github.com/camenduru/gaussian-splatting这一行他这次幸运的把两个子模块都下载下来了之前也会出现网络问题导致没下载下来子模块 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.> 如果下载不行的话可以试试https://github.com/graphdeco-inria/diff-gaussian-rasterization和https://gitlab.inria.fr/bkerbl/simple-knn 这是我从git上复制下来的两个版本链接 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.>

https://github.com/graphdeco-inria/diff-gaussian-rasterization.git 和 https://gitlab.inria.fr/bkerbl/simple-knn.git 可能需要你复制这个链接到浏览器

kirikunkun commented 1 week ago

很感谢，但是这两个链接我打不开，请问方便再发一个嘛！感谢感谢！！

我刚试了一下现在最新的代码最新版的diff-gaussian在这里https://github.com/graphdeco-inria/diff-gaussian-rasterization/tree/9c5c2028f6fbee2be239bc4c9421ff894fe4fbe0 如果你要用最新版的话可以下载这个但是要适配最新版的主代码

graphdeco-inria / gaussian-splatting

Very bad training results #1013