search

dridri / OpenMaxIL-cpp

OpenMax IL C++ wrapper for RaspberryPi
MIT License
23 stars 5 forks source link

Memory leak when destructing #5

Open cedricve opened 6 years ago

cedricve commented 6 years ago

Hey @dridri,

I've been profiling your library with this library https://github.com/MilhouseVH/bcmstat. And it looks that when I destruct and construct the components I got a gpu leak. Any ideas?

Thanks @dridri

screen shot 2017-09-13 at 17 20 25

cedricve commented 6 years ago

Hmm another interesting issue occured, I think it's new (but maybe related to previous issue #1 ). For some reason after a lot of recordings the executable crashed and tried to restart, but its stuck. Although it looks that it's not a GPU memory issue. I have to reboot the Pi to make it work again.

Memory consumption at the moment of locking.

screen shot 2017-09-14 at 21 25 08

The system is locked here:

screen shot 2017-09-14 at 21 26 13

Though the system is more reliable then before, it gets locked after a lot of recordings happend. I'm thinking that this may be caused due to the preview thread not being able to process fast enough? It happened when the system was recording a lot, so it might be a good guess.

More interesting would be to recover the library without the need to have a hard reboot. Any ideas?

Regards Cédric

dridri commented 6 years ago

I will take a look at the leak on destruction. For the second problem I don't see what can cause this since the encoder is constantly running even when not recording, so there is no change in the state of the VideoEncode component (or did you changed that?) thus no memory alloc/dealloc. We cannot recover since it's a GPU lock, and I never found a way to reset the GPU driver when locked

cedricve commented 6 years ago

Thanks for looking into.

For the second point, I did some stress tests and it looks when the system is under pressure (100% a couple of minutes) the lock occurs. I know we have been talking about it and added the high priority thread, but apparently this is still not enough. Not sure if this is something we can do about at all.

Cédric

cedricve commented 6 years ago

Fixed the second point by making sure the CPU doesn't run at 100% a long time. (sleeps here and there)

dridri commented 6 years ago

hrmpf, that's a tricky fix Do you use a sleep/usleep function or pthread_yield ?

Still nothing for the memory leak, I'm starting to think that it's a firmware bug..

cedricve commented 6 years ago

Well not in the preview thread, I'm running FFMPEG in the background to convert h264 inside a MP4 container. This thread is checking a directory of .h264 files and converts them one by one. However if there are a lot of h264 files in this directory, FFMPEG will consume 100% CPU for a long time. This is where I've added a sleep function, so that it doesn't continuously converts.

https://github.com/kerberos-io/machinery/commit/98ede32982a37a8045717b169fcfc7f85221ecb8