OOM when training

[Wastoon]

作者你好！实在是非常棒的工作！我在custom dataset上训练时遇到一些问题，关于显卡内存的，不知道您有一些建议吗？ 具体是在单机多卡训练时，显卡占用不是很均衡，并且占用量也不是很稳定，常常是在几个epoch训练结束后，validation结束后会OOM，关于这个，您有好的建议吗？非常感谢！

[TQTQliu]

Hello, thanks for your interest in our work. Here are some things that might be worth trying: 1) Reduce the batchsize; 2) Reduce the resolution of the image during training here; 3) The extremely large scale of Gaussians may also lead to excessive memory consumption, you can try modifying this to scale_out = torch.clamp_max(self.scale_head(x), 0.01) to constrain the scale. This strategy is used in GPS-Gaussian.

[Wastoon]

Hello, thanks for your interest in our work. Here are some things that might be worth trying:

1. Reduce the batchsize;
2. Reduce the resolution of the image during training here;
3. The extremely large scale of Gaussians may also lead to excessive memory consumption, you can try modifying this to scale_out = torch.clamp_max(self.scale_head(x), 0.01) to constrain the scale. This strategy is used in GPS-Gaussian.

非常感谢您的回复和建议！very insight！