Find extremely big pts_loss output when trying to set cudatoolkit=11.2

[Yang-Hao-Lin]

Hi Junhwa, thanks a lot for the awesome work in the field of scene flow :) When l tried to use your loss in my environment setting (cudatoolkit=11.2), the pts_loss(s_3) became extremely big, for example, bigger than 100000. But when I set the version of cudatoolkit as 10.2, the output of pts_loss became normal (usually less than 10). I can not find out the reason. Have you ever met this kind of situation? Again, thanks a lot.

[hurjunhwa]

Hi, thank you for your interest in our work! I think it happened when the disparity decoder is not properly trained, and there can be multiple reasons such as training instability, invalid inputs, etc.. I would first test the pre-trained model using with the version 11.2 and check if the scene flow accuracy matches the baseline's.

Probably doing the unit-tests of cuda-dependent modules can be also necessary. Please try to use the python implementation of the correlation layer (--correlation_cuda_enabled=False) and also check if the softsplat works well.

If it still doesn't work, please let me know!

[Yang-Hao-Lin]

Hi Junhwa. Thank a lot for your attention! The server administrator of my lab updated my GPU from Tesla P40 to RTX A40 yesterday. I test in the environment setting of cudatoolkit=11.2 again, and now the output of pts_loss is within a reasonable range. So, now the situation is:

Test on GPU P40, cudatoolkit=10.2 -> pts_loss is within a reasonable range; Test on GPU P40, cudatoolkit=11.2 -> pts_loss is extremely big, either correlation_cuda_enabled equals False or equals True, even on a supervised pretrained model; Test on GPU A40, cudatoolkit=11.2 -> pts_loss is within a reasonable range.

It is weird, but there seems to be some compatibility issue between Tesla P40 and cudatoolkit=11.2.

[hurjunhwa]

Thank you for sharing them!