GPU memory usage keeps increasing even hybridize with static_alloc when used in flask debug mode after mxnet 1.6.0post0.

[kohillyang]

Description

Hello, I'm using flask with mxnet to write a server. Since it is a web app, we want the GPU memory is fully static allocated. However, as the title said, I found the GPU memory usage keeps increasing and then raise a OOM when the version of mxnet is 1.6.0post0 and 1.7.0, and if you are using mxnet 1.5.1, then all things are good. Since Flask debug mode uses multi-threading, I think it may be caused by some calls which are not thread-safe.

To Reproduce

This is a naive fLask server:

import mxnet as mx
import os
os.environ["MXNET_CUDNN_AUTOTUNE_DEFAULT"] = "0"
os.environ["MXNET_GPU_MEM_POOL_TYPE"] = "Round"

class Predictor(object):
    def __init__(self):
        ctx = mx.gpu(0)
        net = mx.gluon.model_zoo.vision.resnet50_v1()
        net.initialize()
        net.collect_params().reset_ctx(ctx)
        net.hybridize(active=True)
        max_h = 768
        max_w = 768
        _ = net(mx.nd.zeros(shape=(1, 3, max_h, max_w), ctx=ctx))
        self.ctx = ctx
        self.net = net

    def __call__(self, *args, **kwargs):
        max_h = 768
        max_w = 768
        x_h = np.random.randint(100, max_h)
        x_w = np.random.randint(100, max_w)
        xx = np.random.randn(1, 3, x_h, x_w)
        y = self.net(mx.nd.array(xx, ctx=self.ctx))
        return y.asnumpy().sum()

if __name__ == '__main__':
    import flask
    import tornado.wsgi
    import tornado.httpserver
    import os
    import cv2
    import numpy as np
    from flask_cors import CORS
    import os
    import cv2
    import json
    import logging
    import base64

    os.environ["MXNET_CUDNN_AUTOTUNE_DEFAULT"]="0"
    DEBUG = True
    PORT = 21500
    app = flask.Flask(__name__)
    CORS(app, supports_credentials=True)
    predictor = Predictor()

    @app.route('/test', methods=['POST'])
    def net_forward():
        try:
            r = predictor()
            return None
        except Exception as e:
            logging.exception(e)
            print("failed")
            return flask.jsonify(str(e)), 400

    print("starting webserver...")
    if DEBUG:
        app.run(debug=True, host='0.0.0.0', port=PORT)
    else:
        http_server = tornado.httpserver.HTTPServer(
            tornado.wsgi.WSGIContainer(app))
        http_server.listen(PORT, address="0.0.0.0")
        tornado.ioloop.IOLoop.instance().start()

And just run the following code to request the server:

import base64
import json
import time
import os
import numpy as np
import cv2

def remote_call(url):
    register_data = {"Pic": "xx"}
    data = json.dumps(register_data)
    import requests
    return requests.post(url, data)

if __name__ == '__main__':
    import glob
    import matplotlib.pyplot as plt
    while True:
        register_url = 'http://127.0.0.1:21500/test'
        while True:
            try:
                remote_call(register_url)
            except Exception as e:
                print(e)

Environment

I'm using flask 1.0.2 and tornado 5.1, but I think it is independent of the versions of flask and tornado. We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:

curl --retry 10 -s https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py | python

paste outputs here

/data2/kohill/anaconda3/bin/python /data2/kohill/mx-detection/diagnose.py
----------Python Info----------
Version      : 3.7.0
Compiler     : GCC 7.2.0
Build        : ('default', 'Jun 28 2018 13:15:42')
Arch         : ('64bit', '')
------------Pip Info-----------
Version      : 20.2.2
Directory    : /data2/kohill/anaconda3/lib/python3.7/site-packages/pip
----------MXNet Info-----------
Version      : 1.7.0
Directory    : /data2/kohill/anaconda3/lib/python3.7/site-packages/mxnet
Commit Hash   : 64f737cdd59fe88d2c5b479f25d011c5156b6a8a
64f737cdd59fe88d2c5b479f25d011c5156b6a8a
64f737cdd59fe88d2c5b479f25d011c5156b6a8a
64f737cdd59fe88d2c5b479f25d011c5156b6a8a
64f737cdd59fe88d2c5b479f25d011c5156b6a8a
64f737cdd59fe88d2c5b479f25d011c5156b6a8a
64f737cdd59fe88d2c5b479f25d011c5156b6a8a
64f737cdd59fe88d2c5b479f25d011c5156b6a8a
64f737cdd59fe88d2c5b479f25d011c5156b6a8a
64f737cdd59fe88d2c5b479f25d011c5156b6a8a
Library      : ['/data2/kohill/anaconda3/lib/python3.7/site-packages/mxnet/libmxnet.so']
Build features:
? CUDA
? CUDNN
? NCCL
? CUDA_RTC
? TENSORRT
? CPU_SSE
? CPU_SSE2
? CPU_SSE3
? CPU_SSE4_1
? CPU_SSE4_2
? CPU_SSE4A
? CPU_AVX
? CPU_AVX2
? OPENMP
? SSE
? F16C
? JEMALLOC
? BLAS_OPEN
? BLAS_ATLAS
? BLAS_MKL
? BLAS_APPLE
? LAPACK
? MKLDNN
? OPENCV
? CAFFE
? PROFILER
? DIST_KVSTORE
? CXX14
? INT64_TENSOR_SIZE
? SIGNAL_HANDLER
? DEBUG
? TVM_OP
----------System Info----------
Platform     : Linux-4.15.0-117-generic-x86_64-with-debian-stretch-sid
system       : Linux
node         : ubuntu
release      : 4.15.0-117-generic
version      : #118~16.04.1-Ubuntu SMP Sat Sep 5 23:35:06 UTC 2020
----------Hardware Info----------
machine      : x86_64
processor    : x86_64
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                48
On-line CPU(s) list:   0-47
Thread(s) per core:    2
Core(s) per socket:    12
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
Stepping:              2
CPU MHz:               1200.672
CPU max MHz:           3300.0000
CPU min MHz:           1200.0000
BogoMIPS:              5002.04
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              30720K
NUMA node0 CPU(s):     0-11,24-35
NUMA node1 CPU(s):     12-23,36-47
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0060 sec, LOAD: 1.4688 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.1272 sec, LOAD: 1.2150 sec.
Error open Gluon Tutorial(cn): https://zh.gluon.ai, <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1045)>, DNS finished in 0.10556268692016602 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0053 sec, LOAD: 1.4548 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0048 sec, LOAD: 11.7945 sec.
Error open Conda: https://repo.continuum.io/pkgs/free/, HTTP Error 403: Forbidden, DNS finished in 0.005016326904296875 sec.
----------Environment----------

[wkcn]

Hi @kohillyang , I think it is not related to MXNet.

When there is a new connection, the library flask will create a new python process to handle the connection, which creates a new copy of MXNet instance predictor.

To validate it, you can print the id of the predictor by ~print(id(r))~ print(id(predictor)) in the function def net_forward():

[kohillyang]

@wkcn but even if flask has created a new process, the GPU memory should be freed once the process ends. And the predictor is created in the main function, which should only be called once and has only one predictor instance. On the other side, if the main process has initialized a CUDA environment, the mxnet in the subprocess will fail when inference because their CUDA file descriptor can not be shared between the main process and the sub-process.

BTW. , the pid of the process and the id of the predictor remain unchanged. I print them using the following codes:

        print(id(self))
        print(os.getpid())

PS: ctx.empty_cache() is also not thread-safe. If you called it in two threads, the program would crash in some cases.

Thread-safe is of importance because in some time you need to implement a Block with asnumpy, and it is too hard to implement all blocks as HybridBlock and as an asynchronous way. In pytorch it is not a problem because we have DataParallel. It will start a thread for each CPU instance and gather the results, but this operation is not officially supported by mxnet because at least there are something like https://github.com/apache/incubator-mxnet/issues/13199 which need workarounds.

[kohillyang]

@wkcn predictor is created by predictor = Predictor(), not r = predictor(), since its children function __call__ is override. And the memory usage grows slow, it seems that it is because the memory allocated by the line mx.nd.zeros(shape=(1, 3, max_h, max_w), ctx=ctx) is not freed.

[szha]

@leezu could this be related to the issue fixed by #18328 #18363?

[leezu]

@kohillyang so you are creating a new predictor in every HTTP call? Thus yes, a new Block is created in every HTTP call and due to https://github.com/apache/incubator-mxnet/pull/18328 the parameter of the Block won't be deallocated.

https://github.com/apache/incubator-mxnet/pull/18328/files only contains Python changes. Would you like to try applying the changes to your MXNet files and see if the memory leak goes away. Thank you

[kohillyang]

Why do you think I'm creating a new predictor in each call? Apparently there is only one instance for Predictor.

[leezu]

Nevermind, I didn't read your def net_forward() carefully enough

[leezu]

Thus this is unrelated to #18328