OOM problem

[xuxumiao777]

Happy New Year! I use your provided bash script to train dataset bear. It cannot finish the whole training process and ternimates at about 6500 iterations even after i put the image from cuda to cpu. The error is out of memory. I am using a single 4090. Thanks and hope for your support!

[zoomin-lee]

same error :( it terminates at about 4400 iterations.

Happy New Year

[ymq2017]

Hi, our implementation of 3D regularization loss would take up more memory (we use 48G memory GPU in our paper). And we are optimizing this issue. For reducing memory, I will first suggest some simple solutions.

1. Stop growing the number of gaussian earlier by changing this parameter. Or you can reduce the total iteration to e.g. 7K
2. Temporarily turn off 3D reg loss, or reduce sample points, size and neighbours.
3. Reduce the resolution. Note that you also need to reduce the corresponding resolution of mask label generated from DEVA.

Happy New Year!

[Neal2020GitHub]

Hi, our implementation of 3D regularization loss would take up more memory (we use 48G memory GPU in our paper). And we are optimizing this issue. For reducing memory, I will first suggest some simple solutions.

1. Stop growing the number of gaussian earlier by changing this parameter. Or you can reduce the total iteration to e.g. 7K
2. Temporarily turn off 3D reg loss, or reduce sample points, size and neighbours.
3. Reduce the resolution. Note that you also need to reduce the corresponding resolution of mask label generated from DEVA.

Happy New Year!

Hi, authors! I am wondering have you optimized this OOM issue yet so that we could run on a 24G memory GPU? Many thanks!

[dhgras]

Hi, our implementation of 3D regularization loss would take up more memory (we use 48G memory GPU in our paper). And we are optimizing this issue. For reducing memory, I will first suggest some simple solutions.

1. Stop growing the number of gaussian earlier by changing this parameter. Or you can reduce the total iteration to e.g. 7K
2. Temporarily turn off 3D reg loss, or reduce sample points, size and neighbours.
3. Reduce the resolution. Note that you also need to reduce the corresponding resolution of mask label generated from DEVA.

Happy New Year!

Hi, thank you for your great work! I also want to know if you have optimized the OOM problem? I tried these simple solutions you suggested, but they didn't always work. Once the scene is too large and there is too much data, OOM error will occur. Looking forward to your update. Or could you provide some optimization directions?

[dhgras]

Hi, our implementation of 3D regularization loss would take up more memory (we use 48G memory GPU in our paper). And we are optimizing this issue. For reducing memory, I will first suggest some simple solutions.

1. Stop growing the number of gaussian earlier by changing this parameter. Or you can reduce the total iteration to e.g. 7K
2. Temporarily turn off 3D reg loss, or reduce sample points, size and neighbours.
3. Reduce the resolution. Note that you also need to reduce the corresponding resolution of mask label generated from DEVA.

Happy New Year!

How to temporarily turn off 3D regularization loss? And what I want to know is, what is the impact of turning off 3D regularization loss on the final gaussian grouping in your experiment?

[ljh16042]

嗨，我们对 3D 正则化丢失的实现会占用更多内存（我们在论文中使用 48G 内存 GPU）。我们正在优化这个问题。为了减少内存，我将首先提出一些简单的解决方案。

1. 通过更改此参数来停止增加高斯的数量。或者您可以将总迭代次数减少到 7K
2. 暂时关闭 3D reg 丢失，或减少采样点、大小和相邻点。
3. 降低分辨率。请注意，您还需要降低从 DEVA 生成的掩码标签的相应分辨率。

新年快乐！

嗨，我们对 3D 正则化丢失的实现会占用更多内存（我们在论文中使用 48G 内存 GPU）。我们正在优化这个问题。为了减少内存，我将首先提出一些简单的解决方案。

1. 通过更改此参数来停止增加高斯的数量。或者您可以将总迭代次数减少到 7K
2. 暂时关闭 3D reg 丢失，或减少采样点、大小和相邻点。
3. 降低分辨率。请注意，您还需要降低从 DEVA 生成的掩码标签的相应分辨率。

新年快乐！

Hello, may I ask what model of 48GGPU you are using? I also noticed in your paper that you mentioned using an A00GPU. Could you clarify if it's the 40G version or the 80G version?