Closed elliebirbeck closed 5 years ago
Hey, this is the MXNet Label Bot. Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it. Here are my recommended labels: Bug, Performance
@mxnet-label-bot add [Bug, Performance]
@elliebirbeck copying my answer from the forum here:
This very likely due to you adding ops faster than MXNet is able to process them, the images will be loaded in memory ndarray = mxnet.image.imdecode(imgbuf, to_rgb=1)
and passed to the network.
MXNet is fundamentally asynchronous, it runs on eager execution.
When you call forward, you effectively say, compute this operation as soon as possible. The python callbacks returns almost instantly which allows very simple and intuitive parallelism.
To properly benchmark your code, you need to add a synchronous call:
For example mx.nd.waitall()
or labels.wait_to_read()
or bboxs.asnumpy()
etc
Closing this issue, let's continue the discussion on the forum: https://discuss.mxnet.io/t/memory-leak-when-running-cpu-inference/3256/2
Description
when performing model inference on a cpu I am experiencing RSS memory leak (linear increase from 700MB to 10GB+)
Environment info (Required)
I have tested with mxnet compiled using mkldnn as above and also a version compiled using openblas, and with the most recent (unstable) release of 1.5.0b20190220
Minimum reproducible example
Steps to reproduce
Run the above code snippet and the memory use increases. The leak occurs within the last line (the forward pass through the network to obtain predictions).
What have you tried to solve it?
I have tried several garbage collectors, none of which can find new objects being created. I have tested different pretrained models and a custom model I am trying to use. Also tested on multiple versions of mxnet (compiled using both mkldnn and openblas).