search

dmlc / gluon-cv

Gluon CV Toolkit
http://gluon-cv.mxnet.io
Apache License 2.0
5.83k stars 1.22k forks source link

Training slowfast: computer's memory usage increased #1208

Closed chenyihang1993 closed 4 years ago

chenyihang1993 commented 4 years ago

When I used slowfast to train my own dataset, I found that the computer's memory usage increased with time. Is this normal? Looking forward for your reply.

bryanyzhu commented 4 years ago

Yes, it is normal because data loading is very heavy for video model training. Besides, for MXNet, we use async compute, details can be found here, https://d2l.ai/chapter_computational-performance/async-computation.html

Using async can speed up training, but it may cause the problem that input loaded too fast, and can't be deleted in time. So the CPU memory usage will increase with time.

If you have a limited CPU memory, you can manually perform garbage collection more frequently like what I did here,

https://github.com/dmlc/gluon-cv/blob/master/scripts/action-recognition/test_recognizer.py#L269-L271

chenyihang1993 commented 4 years ago

OK, I got it. THX.