darknet.py - Not releasing Memory after detection in running program (Memory Leak)

[CormacAsGaeilge]

I have a python Rest Server that can call darknet.py to run detections and what appears to be happening is when a detection is made, roughly 1.5-2GB of VRAM on out 1080TI is being used but then not released after the detection is finished. I have found older issues in relation to this issue such as #549 or #289 and their solutions are in relation to an older version of the darknet.py code. I suspect the 'free_detections' function may be the location of this issue but I am unsure how to attempt to fix this issue.

I can try to provide more information if required to understand or help with this issue.

I am running darknet with CUDA 9.2 on Ubuntu 18.04

[dionysos4]

As a workaround you can start at each call a new Process in which you execute the "load_net, load_meta and detection function". After the detection the Process terminates and the GPU memory will be free.

[Bruce-Lee-LY]

Hi, @CormacAsGaeilge , I have the same problem as you. I have used a lot of methods but they have no effect. Could you share how you solved this problem? I am running darknet with CUDA 9.0 on TX1 (Ubuntu 16.04, L4T R28.2). Thanks a lot.

[CormacAsGaeilge]

Hi, @CormacAsGaeilge , I have the same problem as you. I have used a lot of methods but they have no effect. Could you share how you solved this problem? I am running darknet with CUDA 9.0 on TX1 (Ubuntu 16.04, L4T R28.2). Thanks a lot.

I ended up writing a separate python script outside of my main code that would take in an image URL and the weights url I wanted to run Yolo with and calls the darknet.py file.
I then used the subprocess module to call the python script in a separate process. I had the other python file print out its results and used the stdout pipe to read in the output back into my main thread.

from subprocess import call, Popen, PIPE, STDOUT

yolo_process = Popen(
        ['python3', 'yolo.py', 'image.png', 'yolo_18000.weights'], stdout=PIPE, stderr=STDOUT)
results = ""
    for line in yolo_process.stdout:
        results += line.decode('utf-8', "replace")

Hope this helps.

[Bruce-Lee-LY]

@CormacAsGaeilge Thanks for your reply, I will try to use your method.

[sanjaykhanal]

I have written a reply on similar issue in following link. https://github.com/pjreddie/darknet/issues/549#issuecomment-488214928

I hope this will solve the issue.

[Remosy]

Reasons of out of memory: 1: did not free "image" and "detection" created by "darknet.py" 2: called too many times of "load_net" and "load_meta"

Solutions for above: 1: The version of my darknet.py has free_detections(dets, num) and free_image(im)included already in the function def detect(net, meta, image, thresh=.5, hier_thresh=.5, nms=.45). Otherwise, consider of @sanjaykhanal solution.

2: Move

net = load_net(b"StateClassifier/cfg/IH25.cfg", b"StateClassifier/cfg/IH25_150000.weights", 0)
meta = load_meta(b"StateClassifier/cfg/IH2.data")

to the "darknet.py" top as global variables. So, when you next time call the functions of darknet.py from outside script, you don't need to load the two big variables again.

After fixed above, my pyCharm workload is around 1.3GB. It does garbage collection.

[anoob09]

Reasons of out of memory: 1: did not free "image" and "detection" created by "darknet.py" 2: called too many times of "load_net" and "load_meta"

Solutions for above: 1: The version of my darknet.py has free_detections(dets, num) and free_image(im)included already in the function def detect(net, meta, image, thresh=.5, hier_thresh=.5, nms=.45). Otherwise, consider of @sanjaykhanal solution.

2: Move

net = load_net(b"StateClassifier/cfg/IH25.cfg", b"StateClassifier/cfg/IH25_150000.weights", 0)
meta = load_meta(b"StateClassifier/cfg/IH2.data")

to the "darknet.py" top as global variables. So, when you next time call the functions of darknet.py from outside script, you don't need to load the two big variables again.

After fixed above, my pyCharm workload is around 1.3GB. It does garbage collection.

Is there any chance that free_image() and free_detections(dets, num) aren't working as well as they should. Also I moved load_net and load_meta to the top as global variables, it didn't work. Though I managed to let my model run for 25+3 frames before crashing by freeing random variables.

[anoob09]

Hi, @CormacAsGaeilge , I have the same problem as you. I have used a lot of methods but they have no effect. Could you share how you solved this problem? I am running darknet with CUDA 9.0 on TX1 (Ubuntu 16.04, L4T R28.2). Thanks a lot.

I ended up writing a separate python script outside of my main code that would take in an image URL and the weights url I wanted to run Yolo with and calls the darknet.py file. I then used the subprocess module to call the python script in a separate process. I had the other python file print out its results and used the stdout pipe to read in the output back into my main thread.

from subprocess import call, Popen, PIPE, STDOUT

yolo_process = Popen(
        ['python3', 'yolo.py', 'image.png', 'yolo_18000.weights'], stdout=PIPE, stderr=STDOUT)
results = ""
    for line in yolo_process.stdout:
        results += line.decode('utf-8', "replace")

Hope this helps.

But loading the model every time you call detect function would increase the time for detection right? You have to load model every time you have to detect.

[anoob09]

I have written a reply on similar issue in following link. #549 (comment)

I hope this will solve the issue.

You are using darknet as a class I guess. Apart from that it's same code right?

[sanjaykhanal]

Yes it's the same code. Few modifications were made for my convenience. You just have to call self.free_image in 'detect_image' method to solve the issue

[Remosy]

Reasons of out of memory: 1: did not free "image" and "detection" created by "darknet.py" 2: called too many times of "load_net" and "load_meta" Solutions for above: 1: The version of my darknet.py has free_detections(dets, num) and free_image(im)included already in the function def detect(net, meta, image, thresh=.5, hier_thresh=.5, nms=.45). Otherwise, consider of @sanjaykhanal solution. 2: Move

net = load_net(b"StateClassifier/cfg/IH25.cfg", b"StateClassifier/cfg/IH25_150000.weights", 0)
meta = load_meta(b"StateClassifier/cfg/IH2.data")

to the "darknet.py" top as global variables. So, when you next time call the functions of darknet.py from outside script, you don't need to load the two big variables again. After fixed above, my pyCharm workload is around 1.3GB. It does garbage collection.

Is there any chance that free_image() and free_detections(dets, num) aren't working as well as they should. Also I moved load_net and load_meta to the top as global variables, it didn't work. Though I managed to let my model run for 25+3 frames before crashing by freeing random variables.

I used the "darknet.py" in another script by: from FOLDERNAME import darknet

[anoob09]

So I found the solution. My code was crashing after 25 frames. free_image() and free_detections() functions were included in the darknet.py file. After 25 frames, my code was throwing double free or corruption (!prev) Process finished with exit code 134 (interrupted by signal 6: SIGABRT) error. This occurs when you are trying to free some memory which has already been freed I guess. So I removed the free_image() and free_detections(). The code works fine now. So it means that memory is being freed somewhere else. I don't know where the memory is being freed. I don't even know if my assumptions are correct.