Memory leaks after evaluation

[patriciamdr]

Hi!

I am observing CPU RAM memory leaks when training RT-DETR with PyTorch on multiple GPUs after the first evaluation. After that, the RAM usage increased from ~30GB to ~100GB, and continously increases further (and slows down training due to using swapfile). Do you have any ideas what could be the reason for this?

Thanks!

[lyuwenyu]

I rerun rtdetr-r50 on coco dataset, there is a similar phenomenon to what you said, but my machine has 600G ram. I don't know what problem caused it yet.

epoch RAM(G)
01     66
15     114
36     125
42     119