search

CoinCheung / pytorch-loss

label-smooth, amsoftmax, partial-fc, focal-loss, triplet-loss, lovasz-softmax. Maybe useful
MIT License
2.17k stars 374 forks source link

partial_fc_amsoftmax memory leak #33

Closed tianxingyzxq closed 2 years ago

tianxingyzxq commented 2 years ago

partial_fc_amsoftmax memory leak @CoinCheung

tianxingyzxq commented 2 years ago

RuntimeError: CUDA out of memory, after several steps, have you test it?

tianxingyzxq commented 2 years ago

my torch vision is 1.8.1, optimizer is AdamW. sgd has the same problem

CoinCheung commented 2 years ago

It is not likely to be leakage problem since I only use pytorch native operators(I did not write cuda kernels for it). Besides, I myself is training my model with it now, and I observe no such problem.

It is also not likely to be problem of optimizers. Would you please provide detail configuration of your platform and description of how to reproduce it?

CoinCheung commented 2 years ago

Have you tried to use a small batch size and did you observe memory usage is increasing every iteration？

tianxingyzxq commented 2 years ago

I try to reproduce it, and find there is no memory leak at all. My origin train script is wrong. sorry to bother you.