Closed AlexAbades closed 10 months ago
Hello, I figure out what was happenig.
The prolem was that the data was too heacy using the default configuration. If you are using a batch size of 32, which is the default, you will need more then 80GB of memory in the GPU. right now I am using a batch size of 16 and the memory in the GPU is arround 43 GB.
Hope that this helps to someone.
Hello,
I am trying to replicate the results from Pose3D. I am running the code in an high performace computer (HPC), where I have multiple nodes with high RAM avaliable. My sh configuration is:
Which gives me the following resources:
The output error that I am facing is the following:
I have tried to change the max_split_size_mb with the following line in the jobscript.sh (before python run.. ) : export PYTORCH_CUDA_ALLOC_CONF="max_split_size_mb=512"
Although I am reaching the same problem, I can see that the memory usage, average and mean are increasing: Resource usage summary:
Nevertheless, when increasing the value too much it drops again... export PYTORCH_CUDA_ALLOC_CONF="max_split_size_mb=1024" Resource usage summary:
Therefore, I am not sure if this might be the problem...
Aditionally, (I don't know if relevant) before changing the max_split_size_mb, I thought that it was a problem from resources in the HPC so tried to run it multiple times. One thing that I notieced it's that the allocated memory was not always the same:
I am in the right path? Is the fragmentation causing the problem? Is there a way to know the exact number it should be there?
Thank you very much,
Best Regards,
Alex Abades