Killed in the middle of acquisition, memory leak

[yannickbt64]

We're a medical company using Alvium 1800 C-2050c for inspection on production lines.

I'm developing a new software using VimbaPython.

We need to use continuous acquisition most of the time, and make captures (for stocking on network) in singleframe once in awhile. Which is what I'm doing. Except I'm running into an issue.

I joined an MWE showing the issue, stripped down to the basics of what I'm doing.

"""BSD 2-Clause License

Copyright (c) 2019, Allied Vision Technologies GmbH
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this
   list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice,
   this list of conditions and the following disclaimer in the documentation
   and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""

import sys
from typing import Optional, Tuple
from vimba import *
import time

import PyQt5 as qt
#from PyQt5.QtGui import QImage
#from PyQt5.QtCore import Qt, QObject, QThread, pyqtSignal, pyqtSlot, QTimer, QCoreApplication, QMutex, QWaitCondition, QMutexLocker

def print_preamble():
    print('///////////////////////////////////////////')
    print('/// Vimba API Asynchronous Grab Example ///')
    print('///////////////////////////////////////////\n')

def print_usage():
    print('Usage:')
    print('    python asynchronous_grab.py [/x] [-x] [camera_id]')
    print('    python asynchronous_grab.py [/h] [-h]')
    print()
    print('Parameters:')
    print('    /x, -x      If set, use AllocAndAnnounce mode of buffer allocation')
    print('    camera_id   ID of the camera to use (using first camera if not specified)')
    print()

def abort(reason: str, return_code: int = 1, usage: bool = False):
    print(reason + '\n')

    if usage:
        print_usage()

    sys.exit(return_code)

def parse_args() -> Tuple[Optional[str], AllocationMode]:
    args = sys.argv[1:]
    argc = len(args)

    allocation_mode = AllocationMode.AnnounceFrame
    cam_id = ""
    for arg in args:
        if arg in ('/h', '-h'):
            print_usage()
            sys.exit(0)
        elif arg in ('/x', '-x'):
            allocation_mode = AllocationMode.AllocAndAnnounceFrame
        elif not cam_id:
            cam_id = arg

    if argc > 2:
        abort(reason="Invalid number of arguments. Abort.", return_code=2, usage=True)

    return (cam_id if cam_id else None, allocation_mode)

def get_camera(camera_id: Optional[str]) -> Camera:
    with Vimba.get_instance() as vimba:
        if camera_id:
            try:
                return vimba.get_camera_by_id(camera_id)

            except VimbaCameraError:
                abort('Failed to access Camera \'{}\'. Abort.'.format(camera_id))

        else:
            cams = vimba.get_all_cameras()
            if not cams:
                abort('No Cameras accessible. Abort.')

            return cams[0]

def setup_camera(cam: Camera):
    with cam:
        # Try to adjust GeV packet size. This Feature is only available for GigE - Cameras.
        try:
            cam.GVSPAdjustPacketSize.run()

            while not cam.GVSPAdjustPacketSize.is_done():
                pass

        except (AttributeError, VimbaFeatureError):
            pass

def frame_handler(cam: Camera, frame: Frame):
    print('{} acquired {}'.format(cam, frame), flush=True)

    cam.queue_frame(frame)

# def convertImage(image: Frame):
#     (width, height) = (image.get_width(), image.get_height())

#     qtPixFmt = QImage.Format_RGB888
#     if not qtPixFmt:
#         #print(f"AlliedVisionCamDriver.convertImage: Can't find pixel format of image (SN = {self.serialNumber})")
#         return None

#     bytesPerLine = int(image.get_buffer_size()/height)
#     print("BYTESPERLINE", bytesPerLine, "HEIGHT", height, "WIDTH", width)

#     qimg = QImage(image.get_buffer(), width, height, bytesPerLine, qtPixFmt)
#     return qimg

def main():
    print_preamble()
    cam_id, allocation_mode = parse_args()

    sgframes = []

    with Vimba.get_instance():
        with get_camera(cam_id) as cam:

            setup_camera(cam)
            print('Press <enter> to stop Frame acquisition.')

            for i in range(0, 100):
                # Start Streaming with a custom a buffer of 10 Frames (defaults to 5)
                #cam.AcquisitionMode.set('Continuous')
                #cam.ExposureAuto.set(True)
                cam.start_streaming(handler=frame_handler, buffer_count=15, allocation_mode=AllocationMode.AnnounceFrame)
                time.sleep(5)
                cam.stop_streaming()
                #cam.AcquisitionMode.set('SingleFrame')
                frame = cam.get_frame(timeout_ms = 2000)
                #qimg = convertImage(frame)
                #sgframes.append(qimg)

if __name__ == '__main__':
    main()

Running this, I obtain :

Clearly, changing the value of 'buffer_count' impacts how soon the crash happens, but shows there is some kind of memory leak. I suspect the frame buffer is not freed by the garbage collector fast enough, so each call to "start_streaming()" probably creates a new frame buffer, and the memory runs short.

Tested on Ubuntu 20.04 on VirtualBox, on Python 3.8.10. VM has 4GB memory. Syslog says it killed python3 because it ran out of memory. I wouldn't mind help on this issue from AlliedVision. If we can't correct this I'm afraid we'll discontinue the use of AV cameras (which we otherwise like).

I hope you don't tell me I should not to "start_streaming" and "stop_streaming" multiple times within the same with: block. Because if it's the case, this is an awful fix. Each time we re-open the camera, that causes a delay. A delay we want to avoid at all costs in a production setting.

Thanks

[NiklasKroeger-AlliedVision]

Your assumption about memory growing due to frames being allocated at stream start is correct. VimbaPython aims to be as simple to use as possible so it takes care of these tasks, that in lower-level APIs would be done by the user. This prevents you from easily reusing frames from one stream to another and is likely the cause of your growing memory use.

If you want to trigger the garbage collection manually to free the previously allocated frames, you can do so with the Python garbage collector interface. For some examples on how it could be used in situations similar to use you can take a look at #17.

The general idea is to import the garbage collector interface and use gc.collect() in those places, where you want the garbage collector to run explicitly. That call will block until the collection is finished. You could probably replace your call to time.sleep with this and achieve a similar result to what you currently have.

A similar issue was encountered in #60 and the solution suggested there might be of interest, as in that issue the user also tried to implement different "stages" of behaviour in their program. Maybe the approach suggested there is something you could also implement. This might even allow you to use a single start_streaming call reducing the problem of increasing memory due to frame allocation altogether. In that case the recorded images are triggered with the help of software triggering. This is a convenient way to get the desired number of frames for each stage. From your code I can for example see, that you want to only record a single frame after recording a larger batch of frames. This seems like something a well thought out software triggering mechanism should be able to handle.

If you could provide some more insights in the desired sequence of images that you want to record I might be able to provide a more concrete code example for you to work with.

If you do not feel comfortable sharing your requirements publicly here, you can also contact our support via the form on our website and they will be able to help you with getting a good code structure for the task you want to achieve.