search

lyuwenyu / RT-DETR

[CVPR 2024] Official RT-DETR (RTDETR paddle pytorch), Real-Time DEtection TRansformer, DETRs Beat YOLOs on Real-time Object Detection. 🔥 🔥 🔥
Apache License 2.0
2.56k stars 298 forks source link

Out of memory #449

Open waterljwant opened 1 month ago

waterljwant commented 1 month ago

I encountered an out-of-memory issue while training on the COCO dataset using rtdetr_pytorch. The memory usage (not GPU memory) increases gradually with each step. After 2 epochs, it runs out of 188GB memory. Here I show the memory usage at intervals of several steps during the first epoch of training.

image image

image

waterljwant commented 1 month ago

Any suggestions would be greatly appreciated