When using flow loss, the training time increases significantly in the 4D reconstruction task

[tb2-sy]

Thanks for your nice work! I tried to integrate flow loss into 4D-GS, but found that the training time increased significantly when executing the loss.backward() operation, from the original one hour to the current estimated training time of 100 hours, which is almost 100 times longer. Is this reasonable?

[Zerg-Overmind]

Hi, training time should increase but definitely not that slow in our experiments. Could you please let me know the resolution, the number of K per image and your batch size?

[tb2-sy]

The resolution is 1920*1080, the batch size is 1, and K is the default 20.

[Zerg-Overmind]

superized...but thanks for letting me know and I will try the code anyway..

[tb2-sy]

Is this because the resolution is too high?

[Zerg-Overmind]

I don't think so since we did experiments on dynerf dataset whose resolution is also 1k or higher.

[tb2-sy]

Okay, this is a bit weird.

[tb2-sy]

Checking the time consumed by each part of the backward process, we found some problems. The backward process with flow loss consumes 30+ seconds. And between https://github.com/Zerg-Overmind/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L419 to line 448 (Time for RasterizeGaussiansBackwardCUDA2). The time consumption is thousands of times that of https://github.com/Zerg-Overmind/diff-gaussian-rasterization/blob/main/cuda_rasterizer/backward.cu#L756 to line 783 (Time for renderCUDA). As shown in the figure above, when flow loss is involved, cuda backward will be executed twice. The difference between the two times when flow loss is involved is very large. Why is this?

[Xharlie]

30 times is crazy btw, i think there must be something wrong, forgive us not being able to debug on your server, we can only verify everything on our end. Maybe the cuda/gpu version is very different? 30 times sounds like gpu is not properly used.

[penghaohsu0222]

Hi @tb2-sy , I'm trying to do the same thing to integrate the flow loss to 4DGS, but the flow I rendered looks quite weird. Did you successfully integrate it into 4DGS codebase?

[zdrac]

Hi @tb2-sy , @penghaohsu0222 , I'm also trying to integrate the flow loss to 4DGS. Have you solved the problem of slow training on 4DGS?

[120829]

Hi @tb2-sy , @penghaohsu0222 , I'm also trying to integrate the flow loss to 4DGS. Have you solved the problem of slow training on 4DGS?你好，，我也在尝试将流量损失集成到 4DGS 中。 4DGS训练慢的问题解决了吗？

Hello, have you solved this problem? I'm also having the same issue.

[Mercerai]

have you ever solved this issue? I also encounter the same problem. When adopting the flow loss. the backward has become very very slow. @Zerg-Overmind