I am writing this issue regarding to a suspect of memory leak. I am writing some code to run inference of Faster R-CNN with 3 set of model parameters for every image. When I repeatedly execute the inference function within the same process, the number of object grows steadily, while expecting the unused object will be cleared after function execution.
Expected results
1) workspace.ResetWorkspace will remove all objects created, and gc.collect() will free the memory.
2) Back references could be found for all objects created.
Actual results
Some objects are not cleared after the execution of inference function ("GenerateProposalsOp", "CollectAndDistributeFpnRpnProposalsOp"), particularly created by the model creation in 'test_engine.py'
model = model_builder.create(cfg.MODEL.TYPE, train=False, gpu_id=gpu_id)
No back references of "CollectAndDistributeFpnRpnProposalsOp" found by objgraph after inference with 1, 2 and 3 models, with the following 2 lines of code:
obj = objgraph.by_type("CollectAndDistributeFpnRpnProposalsOp")objgraph.show_backrefs(obj, max_depth=10, filename="collect_fpn_obj" + str(i) + ".png")
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
import os
import cv2 # NOQA (Must import before importing caffe2 due to bug in cv2)
import gc
import argparse
import objgraph
import detectron.utils.c2 as c2_utils
from caffe2.python import workspace
from detectron.core.config import assert_and_infer_cfg
from detectron.core.config import cfg
from detectron.core.config import merge_cfg_from_file
from detectron.utils.io import cache_url
from detectron.utils.setup_logging import setup_logging
import detectron.core.test_engine as infer_engine
c2_utils.import_detectron_ops()
# OpenCL may be enabled by default in OpenCV3; disable it because it's not
# thread safe and causes unwanted GPU memory allocations.
cv2.ocl.setUseOpenCL(False)
def parse_args():
parser = argparse.ArgumentParser(description='End-to-end inference')
# ADD SOME ARGUMENT HERE, DOES NOT AFFECT THE GROWTH OF OBJECTS #####
return parser.parse_args()
def inference(args):
workspace.GlobalInit(['caffe2', '--caffe2_log_level=0'])
merge_cfg_from_file(args.cfg)
cfg.NUM_GPUS = 1
args.weights = cache_url(args.weights, cfg.DOWNLOAD_CACHE)
assert_and_infer_cfg(cache_urls=False, make_immutable=False)
assert not cfg.MODEL.RPN_ONLY, \
'RPN models are not supported'
assert not cfg.TEST.PRECOMPUTED_PROPOSALS, \
'Models that require precomputed proposals are not supported'
model = infer_engine.initialize_model_from_cfg(args.weights)
# MODEL INFERENCE HERE, DOES NOT AFFECT THE GROWTH OF OBJECTS #####
workspace.ResetWorkspace('.')
gc.collect()
def main(args):
model_names = ["model0", "model1", 'model2']
config_dir = "DIRECTORY_TO_CONFIGS"
weight_dir = "DIRECTORY_TO_WEIGHTS"
f = open("mem_leak_suspect.txt", "w")
objgraph.show_most_common_types(limit=30, file=f)
f.writelines("\n")
for i in range(len(model_names)):
args.cfg = os.path.join(config_dir, 'config_' + model_names[i] + '.yaml')
args.weights = os.path.join(weight_dir, model_names[i] + '.pkl')
inference(args)
f.writelines("\n===== model: {} =====\n".format(str(i)))
objgraph.show_growth(limit=30, file=f)
obj = objgraph.by_type("GenerateProposalsOp")
objgraph.show_backrefs(obj, max_depth=10, filename="proposalop_obj" + str(i) + ".png")
obj = objgraph.by_type("CollectAndDistributeFpnRpnProposalsOp")
objgraph.show_backrefs(obj, max_depth=10, filename="collect_fpn_obj" + str(i) + ".png")
f.flush()
f.close()
# POST-PROCESS HERE, BUT DOES NOT AFFECT THE GROWTH OF OBJECTS #####
if __name__ == '__main__':
setup_logging(__name__)
args = parse_args()
main(args)
System information
Operating system: Ubuntu 18.04.2 LTS
Compiler version: gcc version 7.4.0
CUDA version: 10.0
cuDNN version: 7.5.1
NVIDIA driver version: 410.104
GPU models (for all devices if they are not all the same): GeForce GTX 1080 Ti
python --version output: Python 3.6.8 :: Anaconda, Inc.
I am writing this issue regarding to a suspect of memory leak. I am writing some code to run inference of Faster R-CNN with 3 set of model parameters for every image. When I repeatedly execute the inference function within the same process, the number of object grows steadily, while expecting the unused object will be cleared after function execution.
Expected results
1)
workspace.ResetWorkspacewill remove all objects created, andgc.collect()will free the memory.2) Back references could be found for all objects created.
Actual results
Some objects are not cleared after the execution of inference function ("GenerateProposalsOp", "CollectAndDistributeFpnRpnProposalsOp"), particularly created by the model creation in 'test_engine.py'
model = model_builder.create(cfg.MODEL.TYPE, train=False, gpu_id=gpu_id)No back references of "CollectAndDistributeFpnRpnProposalsOp" found by objgraph after inference with 1, 2 and 3 models, with the following 2 lines of code:
obj = objgraph.by_type("CollectAndDistributeFpnRpnProposalsOp")objgraph.show_backrefs(obj, max_depth=10, filename="collect_fpn_obj" + str(i) + ".png")Similar result for "GenerateProposalsOp":
And here's the growth of objects:
Detailed steps to reproduce
ResNeXt-152-32x8d model downloaded from: https://dl.fbaipublicfiles.com/detectron/ImageNetPretrained/25093814/X-152-32x8d-IN5k.pkl
Configuration file:
Minimal reproducible code: _DIRECTORY_TO_WEIGHTS contains: 'model0.pkl', 'model1.pkl', 'model2.pkl' DIRECTORY_TO_CONFIGS contains: 'config_model0.yaml', 'config_model1.yaml', 'configmodel2.yaml'
System information
python --versionoutput: Python 3.6.8 :: Anaconda, Inc.