I am running the training over COCO dataset using a slightly modified version of DETR. To put my question simply :
Is it possible for the GPU utilization to remain nominal and everything running perfectly fine and suddenly OOM issue being faced ?
To specify additional details, when I start the training, I see good GPU core utilization and nominal GPU memory utilization. I am running the codes on GPU IDs 0,1 and 3 ( so please ignore the output of GPU02 ).
Then all of a sudden at a point, there is an OOM error :
I do not understand that why suddenly 264 MB of allocation becomes a problem ? There is no indication of it in nvtop graphs. For several iterations, the model was working fine and there was no indication that GPU memory was getting full.
Hello,
I am running the training over COCO dataset using a slightly modified version of DETR. To put my question simply :
To specify additional details, when I start the training, I see good GPU core utilization and nominal GPU memory utilization. I am running the codes on GPU IDs 0,1 and 3 ( so please ignore the output of GPU02 ).
Then all of a sudden at a point, there is an OOM error :
I do not understand that why suddenly 264 MB of allocation becomes a problem ? There is no indication of it in nvtop graphs. For several iterations, the model was working fine and there was no indication that GPU memory was getting full.
Could you help with this ?