search

zhiyuanYU134 / MIRETR

18 stars 2 forks source link

when training ROBI dataset CUDA out of memory #4

Closed songjiehan closed 1 month ago

songjiehan commented 2 months ago

Great good job !!!! When i used the code you provided for learning, i found CUDA out of memory in the process of training and testing the ROBI dataset. My graphics card is 3060. Is it normal for suach aproblem to occur? If so, how much video memory does it take to run your code?

zhiyuanYU134 commented 2 months ago

Great good job !!!! When i used the code you provided for learning, i found CUDA out of memory in the process of training and testing the ROBI dataset. My graphics card is 3060. Is it normal for suach aproblem to occur? If so, how much video memory does it take to run your code?

I use NVIDIA 3090Ti(24GB) to train and test the model. For the ROBI dataset, I think that 12GB of graphics memory is enough for training and testing.

songjiehan commented 1 month ago

Thanks for your answer! I try to change some parameters, i found that the sinkhorn_num_iter parameter was the cause of the excess memory and i set it to 10~30. I wonder if i lower this parameter to 30 will have a big impact on training

zhiyuanYU134 commented 1 month ago

Thanks for your answer! I try to change some parameters, i found that the sinkhorn_num_iter parameter was the cause of the excess memory and i set it to 10~30. I wonder if i lower this parameter to 30 will have a big impact on training

Sinkhorn is an iterative algorithm for solving the optimal transportation problem. This method requires a sufficient number of iterations to approximate the optimal solution. Reducing the number of iterations may cause wrong results.