search

genicam / harvesters

Image Acquisition Library for GenICam-based Machine Vision System
Apache License 2.0
520 stars 89 forks source link

video display delay when using Basler Ace2 USB #370

Open jesperstald opened 2 years ago

jesperstald commented 2 years ago

Describe the Issue Doing a live streaming of the feed from a Basler Ace2, the feed will become delayed even when the buffer size is set very low.

To Reproduce Connect to the camera using Baslers own CTI file (Pylon 7.1), set num_buffers = 1 and filled_buffers_to_hold = 1, start in continuous acq mode, display frames with opencv as they are read from the buffer. If the script lags behind, a delay in the stream will be seen - this can be up to several seconds.

Sample Code I can show a piece of code that demonstrates the reported phenomenon:

import numpy as np
import cv2
from harvesters.core import Harvester

h = Harvester()
h.add_file("C:/Program Files/Basler/pylon 7/Runtime/x64/ProducerU3V.cti")
h.update()

ia = h.create(0)
ia.keep_latest = True
ia.num_buffers = 1
ia.num_filled_buffers_to_hold = 1
ia.remote_device.node_map.AcquisitionMode.value = "Continuous"
ia.remote_device.node_map.AcquisitionFrameRate.value = 50
ia.remote_device.node_map.TriggerMode.value = "Off"
ia.start()

while True:
    with ia.fetch(timeout=np.float32(1)) as buffer:
        img_data = buffer.payload.components[0].data.reshape(ia.remote_device.node_map.Height.value, ia.remote_device.node_map.Width.value)
        bgr_frame = cv2.cvtColor(img_data, cv2.COLOR_BAYER_BG2BGR)

        cv2.imshow("frame", bgr_frame)

        # key input
        key = cv2.pollKey()
        if key & 0xFF == ord('q'):
            break

ia.destroy()
cv2.destroyAllWindows()

Expected Behavior I would expect no delay to be present, as the buffer should be overridden constantly with new images.

Screenshots A video demonstrating the delay can be seen here: https://photos.app.goo.gl/ddTpvRCBF8toi5Ha7

Configuration

Reproducibility This phenomenon can be stably reproduced:

Actions You Have Taken

Any help would be greatly appreciated - i am using Harvester for other Gige cameras and am very happy with it, hope a solution is possible.

Kind Regards, Jesper

jcormier commented 2 years ago

Have you tried increasing num_buffers?

Just my two cents but it would seem to me that you would want at least 2 buffers. Since with only 1 buffer you can't be receiving any new frames for whatever time it takes for python to draw the current image and release the buffer. With 2 you can at least be receiving a new one while you draw the previous one.

kazunarikudo commented 2 years ago

@jesperstald As far as I hear about your use case, you would need to configure the GenTL Producer side. The GenTL SFNC defines a feature called StreamBufferHandlingMode and if your GenTL Producer supports a value called NewestOnly then it should be the way to go; if you've been working with another value, OldestFirst, for example, then you may face the reported phenomenon. Hence I recommend contacting Basler first and asking them if theirs supports NewestOnly. If it's supported please feel free to come back. Thanks, Kazunari.

(@jcormier Thanks for your help. I appreciate that.)

jesperstald commented 2 years ago

Thank you @jcormier and @kazunarikudo for the suggestions!

Have you tried increasing num_buffers?

Just my two cents but it would seem to me that you would want at least 2 buffers. Since with only 1 buffer you can't be receiving any new frames for whatever time it takes for python to draw the current image and release the buffer. With 2 you can at least be receiving a new one while you draw the previous one.

I have tried setting both buffer settings up to 10 frames, but this does not seem to have any effect.

@jesperstald As far as I hear about your use case, you would need to configure the GenTL Producer side. The GenTL SFNC defines a feature called StreamBufferHandlingMode and if your GenTL Producer supports a value called NewestOnly then it should be the way to go; if you've been working with another value, OldestFirst, for example, then you may face the reported phenomenon. Hence I recommend contacting Basler first and asking them if theirs supports NewestOnly. If it's supported please feel free to come back. Thanks, Kazunari.

(@jcormier Thanks for your help. I appreciate that.)

My problem is finding the correct place to modify this, 'ia.data_streams[0].node_map.StreamBufferHandlingMode.value = "NewestFirst"' does not work. Also, when i connect to the camera with another software (CVB GenicamBrowser) but using the same producer file, the delay is not present. This could be due to lower overhead in the code such that the buffer never gets filled however.

jesperstald commented 2 years ago

Update: Using the CVB GenICam Browser, it seems that the StreamBufferHandlingMode of the Stream node has a NewestFirst option. However, as with harverster, this seems a read-only value. image

MathijsNL commented 2 years ago

Can you try to do a firmware update? I have had cameras with greyed out values that should be writeable that worked after a firmware update.

Also, with the matrix vision CTI file I can't set the streambufferhandlingmode, I need the CTI file from the manufacturer of the camera to make it work.

jesperstald commented 2 years ago

Hi Mathijs, thanks for the reply!

I've written to Basler support to get a firmware update file.

I am using the CTI file provided by basler in the screenshot i posted. Does this mean that you have a Basler camera and are able to set this property of the Stream Node or directly in the Remote Device Node ? What cameras/firmware do you have ? I am using a2A1920-160ucBAS cameras.

Kind regards, Jesper

MathijsNL commented 2 years ago

@jesperstald I have a Daheng Imaging camera, and I tested if I am only able to change the handlingmode by using the manufacturers CTI with the following code:

ia.data_streams[0].node_map.StreamBufferHandlingMode.value = "NewestOnly"

Let me know if you can write the node after the firmware update. If it still doesn't work can you try to use the Basler pylon viewer pylon-viewer

MathijsNL commented 2 years ago

@jesperstald I also looked at your sample code, and because of how its setup it will probably always fall behind. Code below might solve your issue for now, even without the NewestOnly setting. It will start a thread that will grab images at a desired framerate. If that thread can keep up the displayed frames might be skipped every now and then, but it will always be the latest frame. I don't see any delays when running this on a 1536x2048 camera, although it doesn't display all the frames because I guess cv2.imshow is a little slow? Displaying images at a lower resolution seems to be showing all the frames, although there is nothing implemented in this script to prevent duplicate frames from showing.

# Inspired by https://pyimagesearch.com/2015/12/21/increasing-webcam-fps-with-python-and-opencv/
from threading import Thread

import cv2
from harvesters.core import Harvester

h = Harvester()
gentlpath = "C:/Program Files/Daheng Imaging/GalaxySDK/GenTL/Win64/GxGVTL.cti"
h.add_file(gentlpath)
h.update()
print(h.device_info_list)

class GenicamVideoStream:
    def __init__(self):
        self.ia = h.create()
        self.ia.remote_device.node_map.PixelFormat.value = 'BayerRG8'
        self.ia.remote_device.node_map.AcquisitionFrameRate.value = 34
        self.ia.remote_device.node_map.ExposureTime.value = 5000
        self.ia.data_streams[0].node_map.StreamBufferHandlingMode.value = "NewestOnly"
        self.stopped = False

    def start(self):
        self.ia.start()
        Thread(target=self.update, args=()).start()
        return self

    def update(self):
        while True:
            if self.stopped:
                return
            self.frame = self.read()

    def read(self):
        with self.ia.fetch() as buffer:
            component = buffer.payload.components[0]
            width = component.width
            height = component.height

            bayer = component.data.reshape((height, width))
            img = cv2.cvtColor(bayer, cv2.COLOR_BayerRG2BGR)
        return img

    def stop(self):
        self.stopped = True
        self.ia.destroy()

stream = GenicamVideoStream().start()

while True:
    bgr_frame = stream.read()
    small = cv2.resize(bgr_frame, (0, 0), fx=0.5, fy=0.5)
    cv2.imshow("frame", small)
    # key input
    key = cv2.pollKey()
    if key & 0xFF == ord('q'):
        break

stream.stop()
cv2.destroyAllWindows()
jesperstald commented 2 years ago

Hi Mathijs,

Thanks for the reply and the code! I am not able to set the NewestOnly value as it is read-only for me. When running the code, it does get a few frames but quickly throws the following error:

Exception has occurred: ResourceInUseException
GenTL exception: Requested resource is already in use. (Message from the source: ) (ID: -1004)
  File "\genicam_high_fps_test.py", line 41, in read
    img = cv2.cvtColor(bayer, cv2.COLOR_BAYER_BG2BGR)
  File "\genicam_high_fps_test.py", line 32, in update
    self.frame = self.read()

Which I don't get with equivalent code not using threading. I wonder, does the harverster ImageAcquirer module not already run in a separate thread, perhaps making the additional thread overkill and causing a conflict?

While it still seems that the buffer setting is still not being respected by Harvester, a few changes made the difference such that the script can handle 120+ fps without falling behind:

import time
import numpy as np
import cv2
from harvesters.core import Harvester

h = Harvester()
h.add_file("C:/Program Files/Basler/pylon 7/Runtime/x64/ProducerU3V.cti")
h.update()

ia = h.create(0)
ia.keep_latest = True
ia.num_buffers = 1
ia.num_filled_buffers_to_hold = 1
ia.remote_device.node_map.AcquisitionMode.value = "Continuous"
ia.remote_device.node_map.AcquisitionFrameRateEnable.value = True
ia.remote_device.node_map.AcquisitionFrameRate.value = 120
ia.remote_device.node_map.TriggerMode.value = "Off"
height = ia.remote_device.node_map.Height.value
width = ia.remote_device.node_map.Width.value
ia.start()

while True:
    loopstart = time.time()
    with ia.fetch(timeout=np.float32(1)) as buffer:
        img_data = buffer.payload.components[0].data.reshape(height, width)
        bgr_frame = cv2.cvtColor(img_data, cv2.COLOR_BAYER_BG2BGR)

        cv2.imshow("frame", bgr_frame)

        # key input
        key = cv2.pollKey()
        if key & 0xFF == ord('q'):
            break
    print(f"Loop time: {time.time() - loopstart:.4f} s")

ia.destroy()
h.reset()
cv2.destroyAllWindows()

The only real change was adding the line ia.remote_device.node_map.AcquisitionFrameRateEnable.value = True and not reading image height / width from the remote_device inside the loop. I also discovered that it is vital to do any image transformations (color conversion etc) inside the with scope.

MathijsNL commented 2 years ago

Which I don't get with equivalent code not using threading. I wonder, does the harverster ImageAcquirer module not already run in a separate thread, perhaps making the additional thread overkill and causing a conflict?

The code works for me, strange that you get this error.

Good to hear you are getting your desired framerate now. Good luck!

jesperstald commented 2 years ago

Thanks Mathijs - I am still awaiting firmware files for the cameras, will let you know if this changes anything.

I am still very much interested in understanding why the buffer size setting seem to be ignored in this instance, and perhaps where I can find the actual buffered image data in the harvester structure. I'm guessing that they must be stored somewhere not on the camera?