After 10 Minutes Arapaho program getting killed

prabindh / darknet

Convolutional Neural Networks

http://pjreddie.com/darknet/

Other

122 stars 46 forks source link

After 10 Minutes Arapaho program getting killed #65

Open shaikhibrahim951 opened 6 years ago

shaikhibrahim951 commented 6 years ago

Hi,

I trained my images with tiny yolo and output is coming good but after 10 to 12 minutes in terminal I am getting killed message. I checked my memory usage using top command then I find that available memory getting reduced by some amount and at some point system is getting hanged. I think some memory leak problem is there and program is not clearing that memory after allocation. Note : I am working on NVIDIA TX2 board having 8GB of RAM. Kindly provide some help regarding this issue. Thanks in advance

prabindh commented 6 years ago

Please share the source code being used. Have you confirmed other components (like camera, opencv, or) do not leak memory as well ?

shaikhibrahim951 commented 6 years ago

My camera and opencv is not resposible for memory leak. I checked it. I will share my source code if you find the problem then it will be good.

test.txt

Attached file is test.cpp in txt form

Thanks

prabindh commented 6 years ago

This may be platform specific behaviour. To be on the safe side, I have pushed a long pending patch for a known memory leak (https://github.com/prabindh/darknet/commit/f6f41a1f3a61b801c109ed62643bcfeb891447a9) - please take a look at latest master.

SinghiDivyanshu commented 6 years ago

hey, in my case the leak problem is because of the resize function in ArapahoV2::Detect. in every frame it is leaking around 60KB of memory.

prabindh commented 6 years ago

@SinghiDivyanshu Which platform (ARM or x64, Windows or Linux) are you seeing this on ?

SinghiDivyanshu commented 6 years ago

@prabindh That problem is resolved as i change the width and height in .cfg file so that resize function is not required.

but still their is a 18KB of definite leak per frame and i'm using linux.

prabindh commented 6 years ago

18 kB seems small for a 3-ch image frame. Will have to check further.

SinghiDivyanshu commented 6 years ago

@prabindh ok thanks for the help.

ArpitaSTugave commented 6 years ago

Do we have a solution for this?

prabindh commented 6 years ago

Havent found time to deep debug, but to rule out CV itself, wrote a modified function with separate Mats for each operation. Could you please give it a try and let me know if it helps ?

Replace the function bool ArapahoV2::Detect( const cv::Mat & inputMat, float thresh, float hier_thresh, int & objectCount)

, with the one in the gist below. https://gist.github.com/prabindh/b7875557e736f75a576972de5618aecf

SinghiDivyanshu commented 6 years ago

@ArpitaSTugave In my case main leak (350KB memory) was due to cv::imshow() function for every frame.

prabindh commented 6 years ago

@SinghiDivyanshu - What version of OpenCV dll is being used in your cases ?

SinghiDivyanshu commented 6 years ago

openCV 3.3

prabindh commented 6 years ago

So I think there are 2 issues here:

Leak in resize - Can we have the gist I posted above tested ?
Leak in imshow() and cap.read() - @SinghiDivyanshu How did you resolve that leak ? Create a new image object every time ?

SinghiDivyanshu commented 6 years ago

leak due to resize: I change the width and height in cfg file.

and for imshow() leak: I'm not able to find any solution, so i remove that function.