huge memory leak while using OpenVINO with yolov3

[gitunit]

i have used this instructions to integrate yolov3 into OpenVINO. when im using it for inference, there is a very big memory leak which will lead to a crash after all RAM gets consumed (and all SWAP as well). therefore i never can finish processing the whole video (2 hours long). i have tested it on a machine with 32 gb RAM and appr. the same in SWAP space but due to the memory leak it gets filled up until it crashes.

[brmarkus]

Can you provide more information about your setup, your environment, using CPU/GPU/VPU, which application(s) (in C++, in Python?), which command line parameters, running natively (Ubuntu 18.04?) or within a container? Are you using the prebuilt OpenVINO installation package or compiled DLDT from source? Can you also describe how you measure the memory consumption?

[gitunit]

im using CVAT with OpenVINO 2019R3 (from the OpenVINO website). thus i believe it (CVAT) is using python. i've tested on Ubuntu 16.04 and also on plain Debian

[brmarkus]

Can you try to run the sample "object_detection_demo_yolov3_async" (from Open-Model-Zoo, part of the OpenVINO installation), please? It allows you to specify a video file via the parameter "-i" as well. With "-d" you can specify whether to use CPU, GPU or VPU (e.g. Myriax/MyriadX).

Try to use a "native" OpenVINO sample first - and later maybe file an issue at "https://github.com/opencv/cvat" ;-)

[gitunit]

ok. i've tried and couldn't observe a memory leak. is there an equivalent python version?

[benhoff]

This is probably an issue with CVAT. I think there are prexisting issues that note this issue.

[brmarkus]

There is a Python demo as well: /opt/intel/openvino/inference_engine/demos/python_demos/object_detection_demo_yolov3_async

[gitunit]

thx. where do i get CPU extensions (i need to test on CPU since CVAT only supports that, so i can compare better)? i get the following error: [ ERROR ] Following layers are not supported by the plugin for specified device CPU: detector/yolo-v3/ResizeNearestNeighbor, detector/yolo-v3/ResizeNearestNeighbor_1, detector/yolo-v3/Conv_22/BiasAdd/YoloRegion, detector/yolo-v3/Conv_14/BiasAdd/YoloRegion, detector/yolo-v3/Conv_6/BiasAdd/YoloRegion

[brmarkus]

Under /opt/intel/openvino/inference_engine/lib/intel64/ , you can specifiy e.g. "-l /opt/intel/openvino/inference_engine/lib/intel64/libMKLDNNPlugin.so".

[gitunit]

[ INFO ] Creating Inference Engine... Traceback (most recent call last): File "/opt/intel/openvino_2019.3.376/inference_engine/demos/python_demos/object_detection_demo_yolov3_async/object_detection_demo_yolov3_async.py", line 359, in <module> sys.exit(main() or 0) File "/opt/intel/openvino_2019.3.376/inference_engine/demos/python_demos/object_detection_demo_yolov3_async/object_detection_demo_yolov3_async.py", line 178, in main ie.add_extension(args.cpu_extension, "CPU") File "ie_api.pyx", line 118, in openvino.inference_engine.ie_api.IECore.add_extension RuntimeError: dlSym cannot locate method 'CreateExtension': /opt/intel/openvino/inference_engine/lib/intel64/libMKLDNNPlugin.so: undefined symbol: CreateExtension something missing from the installation maybe?

[brmarkus]

Have you installed the packages in "requirements.txt" under /opt/intel/openvino/inference_engine/demos/python_demos?

I don't need to specify a CPU extension... It just works for me:

$> python3 object_detection_demo_yolov3_async.py -m /var/data/yolo-v3/FP32/yolo-v3.xml -i /var/data/Airport-1080p-30FPS-5Mbps-AVC.mp4 -d CPU --labels /var/data/yolo-v3/FP32/yolo-v3.labels -t 0.6

And also for tiny-yolo-v3: $> python3 object_detection_demo_yolov3_async.py -m /var/data/tiny-yolo-v3/FP32/tiny-yolo-v3.xml -i /var/data/Airport-1080p-30FPS-5Mbps-AVC.mp4 -d CPU --labels /var/data/tiny-yolo-v3/FP32/tiny-yolo-v3.labels -t 0.6

I'm using "Python 3.5.2". I'm using "openvino_2020.1.023".

I think your older version of OpenVINO still has the old CPU-extension library.... Can you check your folder "/opt/intel/openvino/inference_engine/lib/intel64/" and see whether you see "libcpu_extension*.so"... Then use this instead of "libMKLDNNPlugin.so", sorry.

[gitunit]

thx, that did the trick. i used libcpu_extension_avx2.so and the inference is running. no memory leak there so far.

[Maioy97]

@gitunit where do people usually get the cpu extensions people reference them everywhere but there isn't a clear method to get them

[gitunit]

@Maioy97 open model zoo