BUG: Repeatedly opening and closing camera eventually causes error

roundhousemb commented 3 weeks ago

Describe the issue:

In our application we call InstantCamera.Open capture an image and then call InstantCamera.Close after we're done grabbing. After many image captures we would eventually get an error when executing a software trigger that the camera had not started grabbing. Occasionally we would get a different error complaining about hitting an open file limit in our Ubuntu operating system. I was eventually able to track the issue down to the fact that we were re-opening and closing the device for each image capture. It seems like InstantCamera.Close is not cleaning up everything it should be.

Reproduce the code example:

import time
from pypylon import pylon
cam = pylon.InstantCamera(pylon.TlFactory.GetInstance().CreateFirstDevice())
count = 0
while True:
    cam.Open()
    time.sleep(0.5)
    cam.Close()
    if count % 100:
        print(f"Iterations: {count}")

Error message:

The example code above will cause an error after ~950 iterations and will raise this exception

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/home/nbvision/.local/lib/python3.10/site-packages/pypylon/pylon.py", line 3473, in Open
_genicam.RuntimeException: Failed to open 'Basler a2A5320-7gcPRO#00305335F664#192.168.1.205:3956'. The device is not reachable or controlled by another application. Err: *** Error getting status message (0xE1020018) : RuntimeException thrown (file 'GigEDevice.cpp', line 314)

As mentioned before this error changes depending on how I try to access the camera next after hitting the limit.

Is your camera operational in Basler pylon viewer on your platform

Yes

Hardware setup & camera model(s) used

Intel i9 13900k x86 Ubuntu 22.04 LTS 64GB ddr5

There is a single ethernet switch connecting the cameras to the system. I don't know the exact model at this time but I'll try and find it.

Runtime information:

python: 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0]
platform: linux/x86_64/5.15.0-1066-intel-iotg
pypylon: 4.0.0 / 8.0.0.10

roundhousemb commented 3 weeks ago

The error I listed in the original post is what happens when I strip things down to reproduce my issue. However this is the error I initially got. I believe it is related (the camera is not open so grabbing does not get properly started and the software trigger cannot complete):

Traceback (most recent call last):
  File "/home/nbvision/nb.mfg.robotics.staging/./test_camera_error.py", line 48, in <module>
    main()
  File "/home/nbvision/nb.mfg.robotics.staging/./test_camera_error.py", line 26, in main
    cam.simple_run(None, "/home/nbvision/TEST_IMAGE.tiff", settings)
  File "/home/nbvision/nb.mfg.robotics.staging/common/camera.py", line 323, in simple_run
    self._run_camera(camera_image_fn, camera_image_t_fn)
  File "/home/nbvision/nb.mfg.robotics.staging/common/camera.py", line 270, in _run_camera
    self._cam.ExecuteSoftwareTrigger()
  File "/home/nbvision/.local/lib/python3.10/site-packages/pypylon/pylon.py", line 4122, in ExecuteSoftwareTrigger
    return _pylon.InstantCamera_ExecuteSoftwareTrigger(self)
_genicam.RuntimeException: Grabbing has not been started. : RuntimeException thrown (file 'InstantCameraImpl.h', line 1563)

And the following error is what I think is the actual root cause. I can force this error to appear by instead running in emulation mode. However I believe it illustrates the root cause of the issue.

Traceback (most recent call last):
  File "/home/nbvision/nb.mfg.robotics.staging/./test_camera_error.py", line 48, in <module>
    main()
  File "/home/nbvision/nb.mfg.robotics.staging/./test_camera_error.py", line 26, in main
    cam.simple_run(None, "/home/nbvision/TEST_IMAGE.tiff", settings)
  File "/home/nbvision/nb.mfg.robotics.staging/common/camera.py", line 321, in simple_run
    self._run_emulation(camera_image_fn, camera_image_t_fn)
  File "/home/nbvision/nb.mfg.robotics.staging/common/camera.py", line 247, in _run_emulation
    result_emulated = self._cam.RetrieveResult(1000)
  File "/home/nbvision/.local/lib/python3.10/site-packages/pypylon/pylon.py", line 3639, in RetrieveResult
    return _pylon.InstantCamera_RetrieveResult(self, *args)
_genicam.GenericException: WaitObject duplicate failed (0): Too many open files. Reached open files limit: To change the limit permanently on a ubuntu system, edit /etc/security/limits.conf : RuntimeException thrown (file 'WaitObject_Unix.cpp', line 76)

thiesmoeller commented 3 weeks ago

Same as https://github.com/basler/pypylon/issues/776 FileHandle Leak in transportlayer is fixed in upcoming pylon version

As we currently have not released the updated pypylon version you have to compile on your own for now:

Install Pylon 8.0.1: https://www.baslerweb.com/de-de/downloads/software/96111466/

If you use the debian install pylon sdk will be in /opt/pylon

so you can easily install using:

PYLON_ROOT=/opt/pylon pip install git+https://github.com/basler/pypylon.git@feature/pylon_8_0_0

roundhousemb commented 3 weeks ago

Thank you for your response and sorry for missing the duplicate issue. I will try compiling and testing the newer version but may just use a workaround to avoid this situation until the newer pypylon version is released.

Will close this ticket, thanks again for the support.

basler / pypylon