search

bonsai-rx / spinnaker

A Bonsai library for interfacing with FLIR cameras using the Spinnaker SDK
Other
2 stars 4 forks source link

Memory leak #1

Open glopesdev opened 5 years ago

glopesdev commented 5 years ago

Original report by Anonymous.

I was using this plugin today with a 120Hz camera. Worked well for 2 hours or so and then I my workflow started freezing. Looked at RAM usage and it was hitting 100% repeatedly and then falling off on a 20 second or so cycle. I happened to be running in debug mode out of VS so I could profile RAM and VS claimed no responsibility for this. Allocated objects and RAM were stable during this behavior. This might indicate that the master engineers over at FLIR have some huge memory leak in the native API. I know this is a sketchy account but any advice would be appreciated.

glopesdev commented 5 years ago

Original comment by Jonathan Newman (Bitbucket: [Jonathan Newman](https://bitbucket.org/Jonathan Newman), ).

This issue was posted by me BTW. I was not logged in when i made it.

glopesdev commented 3 years ago

Will have to reevaluate this on the new Spinnaker SDK 2.0 API.

Ag8833 commented 3 years ago

Hello glopesdev! Have you had any success with this? I'm seeing the exact same issue. My setup has 2 FLIR Blackfly S USB3 cameras using the Spinnaker API (version 1.29).

Randomly the RAM usage will start to climb and never drops back down. It will continue climbing until I stop it or the system runs out of RAM and the program crashes. I have both my cameras constantly triggering, however I modified them at one point to start/stop triggering at different points in the system, and that seemed to cause this issue to happen more frequently. Now with both cameras constantly triggering I probably see this issue every 1-2 hours or so.

Same thing here with me, when I look in VS I see no excess memory usage, and I can check it at different intervals and the usage of RAM from VS never increases. I've tried using external memory debuggers and various tools but nothing ever points to a memory leak from VS. My program usually runs around 2-3 GB, but when the leak happens it can climb up to around 25 GB, yet VS still shows nothing taking up this much memory.

I've tried changing the way the camera allocates the buffers to only be a singular buffer but this issue still occurs. Odd thing is we have a similar system that is running almost identical code to operate these cameras and doesn't see any issues, however that machine is running version 1.23.

LWitter commented 3 years ago

It seems to also happen to me: 3x IDS cameras running at 160Hz. Memory usage is now at 15GB+, and the I'm not even running acquisition, just in between streams. VERY frequent crashes to the point that it just not works.

LWitter commented 3 years ago

Just ran a very short acquisition again (~5 seconds). While the files are just a few MBs, memory usage has increased to 28GB+ ScreenshotMemory

liadJB commented 2 years ago

Has anyone solved this issue? my code with just a single camera crashes after 20 minutes of recording. At the beginning it crashed with OutOfMemory, when it recorded at 100 Hz. But, when I made the camera run at 20 Hz it still happened. THis is the error I am getting: Bonsai.WorkflowRuntimeException: Insufficient memory: Failed to allocate 1555200 bytesin function cv::OutOfMemoryError, alloc.cpp(52) ---> OpenCV.Net.CVException: Insufficient memory: Failed to allocate 1555200 bytesin function cv::OutOfMemoryError, alloc.cpp(52) at OpenCV.Net.Native.NativeMethods.ErrorExceptionCallback(Int32 status, String func_name, String err_msg, String file_name, Int32 line, IntPtr userData) at OpenCV.Net.Native.NativeMethods.cvCreateImage(Size size, IplDepth depth, Int32 channels) at Bonsai.Spinnaker.SpinnakerCapture.<>cDisplayClass14_0.b0(IManagedImage image) at Bonsai.Spinnaker.SpinnakerCapture.<>cDisplayClass16_1`1.<b1>d.MoveNext() --- End of inner exception stack trace ---

Ag8833 commented 2 years ago

@liadJB Hello! My issue seems a little different than yours but maybe I can help. I did get this fixed but it ended up being a memory leak through some UI element that was causing the memory to continually increase until my software crashed. It seems like if you have any sort of memory issues the garbage collector might not work correctly, causing each frame from the camera to continue to exist in memory. I only caught it by continually outputting my memory usage until I could track down the point in my sequence where the memory leak would start. I would try doing something like that and see if it helps.