alliedvision / VimbaPython

Old Allied Vision Vimba Python API. The successor to this API is VmbPy
BSD 2-Clause "Simplified" License
93 stars 40 forks source link

Batch saving with cv2 for asynchronous image acquisition #62

Open NiklasKroeger-AlliedVision opened 3 years ago

NiklasKroeger-AlliedVision commented 3 years ago

I have a question related to the last framework posted above: Is there a nice and easy way (preferably built in vimba) to stack the 10 frames (and many more optionally) from the software triggering loop and save them as a batch afterwards using cv2.imwritemulti() for example? And if not, how to make sure each frame can get written before the next Software triggering happens?

I ask this because I'm having problems at saving the images with cv2.imwrite() within the frame_callback(). The Triggering goes faster than the writing. In the end several images are missing on disk.

Just to give you an idea it looks as follows:

def take_pic(num_pic=60):

    with Vimba.get_instance() as vimba:
        with vimba.get_all_cameras()[0] as cam:            
            try:
                cam.start_streaming(handler=frame_callback)

                for _ in range(num_pic):
                    cam.TriggerSoftware.run()
                    time.sleep(1)

            finally:
                cam.stop_streaming()

def frame_callback(cam: vimba.Camera, frame: vimba.Frame):

    # Called every time a new frame is received. Perform image analysis here
    frame=do_the_necessary_transformations(frame) 
    cv2.imwrite(some_filename, frame.as_opencv_image())

    # Hand used frame back to Vimba so it can store the next image in this memory
    cam.queue_frame(frame)

Also I'd prefer to remove the 1 sec. delay after each triggering. Many thanks for the material already shared. It helped a lot!!

Originally posted by @frischwood in https://github.com/alliedvision/VimbaPython/issues/60#issuecomment-883559785

NiklasKroeger-AlliedVision commented 3 years ago

Are you certain that you want to use cv2.imwritemulti? I have been playing around with it and from my understanding it is intended to support multi-page Tiff files and nothing else. I personally have not been able to find an image viewer that easily allows me to view these multi-page Tiff files.

To tackle the problem you are describing (imwrite is too slow to be used in the frame callback) I have a different suggestion: create a thread that is responsible for writing out your recorded frames. This way the frame callback does not have to perform the slow disk-io. Instead it only takes the resulting opencv image from the received frame and puts it into a queue from which the thread writing to disk takes them. of course this queue will keep growing if you continuously record new images and these images are recorded faster than they can be written to disk, but if it is only intended to record a certain number of frames I believe this may solve your problem. Here is a code example doing just that:

import functools
import queue
import threading
import time

import cv2
import vimba

class ImageRecorder:
    def __init__(self, cam: vimba.Camera, frame_queue: queue.Queue):
        self._cam = cam
        self._queue = frame_queue

    def __call__(self, cam: vimba.Camera, frame: vimba.Frame):
        # Place the image data as an opencv image and the frame ID into the queue
        self._queue.put((frame.as_opencv_image(), frame.get_id()))

        # Hand used frame back to Vimba so it can store the next image in this memory
        cam.queue_frame(frame)

    def _setup_software_triggering(self):
        # Always set the selector first so that folling features are applied correctly!
        self._cam.TriggerSelector.set('FrameStart')

        # optional in this example but good practice as it might be needed for hadware triggering
        self._cam.TriggerActivation.set('RisingEdge')

        # Make camera listen to Software trigger
        self._cam.TriggerSource.set('Software')
        self._cam.TriggerMode.set('On')

    def record_images(self, num_pics: int):
        # This method assumes software trigger is desired. Free run image acquisition would work
        # similarly to get higher fps from the camera
        with vimba.Vimba.get_instance():
            with self._cam:
                try:
                    self._setup_software_triggering()
                    self._cam.start_streaming(handler=self)
                    for i in range(num_pics):
                        print(i)
                        self._cam.TriggerSoftware.run()
                        # Delay between images can be adjusted or removed entirely
                        time.sleep(0.1)
                finally:
                    self._cam.stop_streaming()

def write_image(frame_queue: queue.Queue):
    while True:
        # Get an element from the queue.
        frame, id = frame_queue.get()
        cv2.imwrite(f'image_{id}.png', frame)
        # let the queue know we are finished with this element so the main thread can figure out
        # when the queue is finished completely
        frame_queue.task_done()

def main():
    num_pics = 60
    with vimba.Vimba.get_instance() as vmb:
        cams = vmb.get_all_cameras()

        frame_queue = queue.Queue()
        recorder = ImageRecorder(cam=cams[0], frame_queue=frame_queue)
        # Start a thread that runs write_image(frame_queue). Marking it as daemon allows the python
        # program to exit even though that thread is still running. The thread will then be stopped
        # when the program exits
        threading.Thread(target=functools.partial(write_image, frame_queue), daemon=True).start()

        recorder.record_images(num_pics=num_pics)
        frame_queue.join()

if __name__ == "__main__":
    main()

I hope this helps. If you have further questions or if this does not solve the problem please let me know.

frischwood commented 3 years ago

Thanks a lot for your answer. It helped modifying the structure (to asynchronous). However it seems the thread stops once the main() exits. Running the code above yields 7 writes while 60 triggers happen. Do you get 60 images written on disk?

NiklasKroeger-AlliedVision commented 3 years ago

That is odd. Yes if I run the example above it writes 60 files (image_0.png ... image_59.png) to the directory where I ran the code.

However it seems the thread stops once the main() exits.

This sounds like the frame_queue.join() is not working.... That line should wait until all frames in the queue have been processed and written to disk. Have you modified the code I pasted or are you running it as in the example above?

frischwood commented 3 years ago

Have you modified the code I pasted or are you running it as in the example above?

Didn't modify a thing... but okay, then it must be on my side.

This sounds like the frame_queue.join() is not working...

Yes indeed, I'm gonna figure that out. I'll let you know if it ends up working. Thanks again!

NiklasKroeger-AlliedVision commented 3 years ago

Hmmm. It really should not make a difference but just for debugging: I used Python 3.8.5 on a Windows 64bit system to run the example above....

NiklasKroeger-AlliedVision commented 3 years ago

Here is a smaller test for the use of queue.Queue independent of VimbaPython. Maybe this will help narrow down why on your machine the queue is not emptied before exiting the program:

import functools
import queue
import threading
import time

def producer(num, frame_queue):
    for i in range(num):
        frame_queue.put(i)
        print(f"PRODUCER: added {i} to queue")
        time.sleep(0.05)

def consumer(frame_queue):
    while True:
        i = frame_queue.get()
        print(f"CONSUMER: popped {i} from queue")
        time.sleep(0.1)
        frame_queue.task_done()

def main():
    frame_queue = queue.Queue()
    threading.Thread(target=functools.partial(consumer, frame_queue), daemon=True).start()
    producer(10, frame_queue)
    print("MAIN: waiting for frame_queue to be emptied")
    frame_queue.join()

if __name__ == "__main__":
    main()

On my machine I get the following output if I run it:

PRODUCER: added 0 to queue
CONSUMER: popped 0 from queue
PRODUCER: added 1 to queue
CONSUMER: popped 1 from queue
PRODUCER: added 2 to queue
PRODUCER: added 3 to queue
CONSUMER: popped 2 from queue
PRODUCER: added 4 to queue
PRODUCER: added 5 to queue
CONSUMER: popped 3 from queue
PRODUCER: added 6 to queue
PRODUCER: added 7 to queue
CONSUMER: popped 4 from queue
PRODUCER: added 8 to queue
PRODUCER: added 9 to queue
CONSUMER: popped 5 from queue
MAIN: waiting for frame_queue to be emptied
CONSUMER: popped 6 from queue
CONSUMER: popped 7 from queue
CONSUMER: popped 8 from queue
CONSUMER: popped 9 from queue
frischwood commented 3 years ago

This last example works well. I can reproduce the same output. Thanks ! I'm still stuck and working on the one with VimbaPython.

NiklasKroeger-AlliedVision commented 3 years ago

That is very odd.... If you need further assistance let me know!

newtop95 commented 3 years ago

I have a similar issue. I've used your provided code and just adapted it so the pixel format is set to Bgr8, that works just fine. The first time i call main its runs perfect, after that i always get errors.

ValueError: 0 is not a valid PixelFormat

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 232, in 'calling callback function'
  File "/usr/local/lib/python3.7/dist-packages/vimba/camera.py", line 979, in __frame_cb_wrapper
    raise e
  File "/usr/local/lib/python3.7/dist-packages/vimba/camera.py", line 971, in __frame_cb_wrapper
    context.frames_handler(self, frame)
  File "/home/pi/Desktop/test.py", line 16, in __call__
    self._queue.put((frame.as_opencv_image(), frame.get_id()))
  File "/usr/local/lib/python3.7/dist-packages/vimba/frame.py", line 877, in as_opencv_image
    str(PixelFormat(self._frame.pixelFormat))))
  File "/usr/lib/python3.7/enum.py", line 310, in __call__
    return cls.__new__(cls, value)
  File "/usr/lib/python3.7/enum.py", line 564, in __new__
    raise exc
  File "/usr/lib/python3.7/enum.py", line 548, in __new__
    result = cls._missing_(value)
  File "/usr/lib/python3.7/enum.py", line 577, in _missing_
    raise ValueError("%r is not a valid %s" % (value, cls.__name__))
ValueError: 0 is not a valid `PixelFormat

Strangely, the error doesn't occur when i delete the saved pictures before calling main the second time, but as soon as the first call of the main function saves images into my directory and i don't delete them after that before running main a second time, the error above occurs...

Here is my code:

import functools
import queue
import threading
import time
from datetime import datetime
import cv2
import vimba

class ImageRecorder:
    def __init__(self, cam: vimba.Camera, frame_queue: queue.Queue):
        self._cam = cam
        self._queue = frame_queue

    def __call__(self, cam: vimba.Camera, frame: vimba.Frame):
        # Place the image data as an opencv image and the frame ID into the queue
        self._queue.put((frame.as_opencv_image(), frame.get_id()))

        # Hand used frame back to Vimba so it can store the next image in this memory
        cam.queue_frame(frame)

    def record_images(self, num_pics: int):
        # This method assumes software trigger is desired. Free run image acquisition would work
        # similarly to get higher fps from the camera
        with vimba.Vimba.get_instance():
            with self._cam:
                try:
                    self._cam.start_streaming(handler=self)
                    for i in range(num_pics):
                        #print(i)
                        self._cam.TriggerSoftware.run()
                        # Delay between images can be adjusted or removed entirely
                        time.sleep(0.03)
                finally:
                    self._cam.stop_streaming()

def write_image(frame_queue: queue.Queue):
    while True:
        # Get an element from the queue.
        frame, id = frame_queue.get()
        cv2.imwrite('/home/pi/Desktop/pictures/test/%s.jpg' %(datetime.now().timestamp()), frame)
        # let the queue know we are finished with this element so the main thread can figure out
        # when the queue is finished completely
        frame_queue.task_done()

def main():
    num_pics = 10
    with vimba.Vimba.get_instance() as vmb:
        cams = vmb.get_all_cameras()
        with cams[0] as cam:

            print(cam.get_pixel_format())
            #cam.load_settings(settings_file, vimba.camera.PersistType.All)
            cam.set_pixel_format(vimba.camera.PixelFormat.Bgr8)

            frame_queue = queue.Queue()
            recorder = ImageRecorder(cam=cam, frame_queue=frame_queue)
            # Start a thread that runs write_image(frame_queue). Marking it as daemon allows the python
            # program to exit even though that thread is still running. The thread will then be stopped
            # when the program exits
            threading.Thread(target=functools.partial(write_image, frame_queue), daemon=True).start()

            recorder.record_images(num_pics=num_pics)
            frame_queue.join()

if __name__ == "__main__":
    main()
newtop95 commented 3 years ago

In addition, i also discovered that a lot of asynchronous captured images are mostly black and incomplete. Do you have any suggestions?

frischwood commented 3 years ago

Have you checked the exposure time settings? This might be a reason for dark pictures. For the incompleteness, I never had this issue with the above framework. You could run a sanity check such as:

if frame.get_status() == FrameStatus.Complete:
      do_something()
NiklasKroeger-AlliedVision commented 3 years ago

In addition, i also discovered that a lot of asynchronous captured images are mostly black and incomplete. Do you have any suggestions?

You are on the right track with this observation. This is what is happening:

The best way to work with this problem is to simply check that the frame you received is actually marked as complete in the frame callback. For this you can use frame.get_status(). Here would be just the updated ImageRecorder.__call__ method which is the only thing that would need to be changed:

    def __call__(self, cam: vimba.Camera, frame: vimba.Frame):
        # Place the image data as an opencv image and the frame ID into the queue
        if frame.get_status() == vimba.frame.FrameStatus.Complete:
            self._queue.put((frame.as_opencv_image(), frame.get_id()))

        # Hand used frame back to Vimba so it can store the next image in this memory
        cam.queue_frame(frame)

EDIT: I just saw that @frischwood suggested this same thing. Thanks for that! :)

Side note: if you are only adding complete frames to your save queue, you might save fewer images than you expect, because you are only executing the software trigger 10 times. So if out of those 10 triggered frames 2 are incomplete, only 8 images would be saved. If you want to make sure, that you always get 10 frames saved to disk, you would need to somehow communicate the fact that a frame was incomplete back to your main thread so that it triggers an additional image for every incomplete frame. Generally however, you should try to fix the root cause for your incomplete frames instead of adding such a brute force workaround.

Possible reasons for incomplete frames:

If you are having more general trouble getting a stable stream with our asynchronous image acquisition feel free to also contact our support via the form on our website. They are very experienced in optimizing system settings for good transfer performance and will be happy to help you.

newtop95 commented 3 years ago

@frischwood and @NiklasKroeger-AlliedVision thanks a lot for your suggestions! The sanity check and limiting the device link throughput (the Pi has USB 3.0 support, but it seems it can't achieve the full speed for whatever reason) helped! It now works reliably.

newtop95 commented 3 years ago

Unfortunately, I have encountered another problem.

I checked the captured images and it seems that my code only saves the first images correctly and then repeats the last saved images.

This is my code:

from cv2 import cv2
import vimba 
from datetime import datetime
import time
import functools
import queue
import threading

#get and parse settings file

import yaml
with open('/home/pi/Desktop/WST_PictureCollectionModuleNew/config.yaml', 'r') as file:
    config = yaml.safe_load(file)

# settings file for camera
settings_file = config['Camera_settings']['settings_file']

#----------------------------------------------------------------------

class ImageRecorder:
    def __init__(self, cam: vimba.Camera, frame_queue: queue.Queue):
        self._cam = cam
        self._queue = frame_queue

    def __call__(self, cam: vimba.Camera, frame: vimba.Frame):
        # Place the image data as an opencv image and the frame ID into the queue
        #sanity check for complete frames
        if frame.get_status() == vimba.frame.FrameStatus.Complete:

            self._queue.put((frame.as_opencv_image(), frame.get_id()))

        # Hand used frame back to Vimba so it can store the next image in this memory
        cam.queue_frame(frame)

    def record_images(self, num_pics: int):
        # This method assumes software trigger is desired. Free run image acquisition would work
        # similarly to get higher fps from the camera
        with vimba.Vimba.get_instance():
            with self._cam:
                try:
                    self._cam.start_streaming(handler=self)
                    for i in range(num_pics):

                        self._cam.TriggerSoftware.run()
                        # Delay between images can be adjusted or removed entirely
                        time.sleep(0.03)

                finally:
                    self._cam.stop_streaming()

def write_image(frame_queue: queue.Queue, path, pic_format):

    while True:
        # Get an element from the queue.
        path = path
        frame, id = frame_queue.get()

        cv2.imwrite(f'/home/pi/Desktop/pictures/%s/{id}.%s' %(path,  pic_format), frame) #datetime.now().timestamp(), pic_format), frame)
        # let the queue know we are finished with this element so the main thread can figure out
        # when the queue is finished completely
        frame_queue.task_done()

def capture_picture(path, pic_format):

    with vimba.Vimba.get_instance() as vmb:
        cams = vmb.get_all_cameras()
        with cams[0] as cam:

            num_pics = 30

            cam.load_settings(settings_file, vimba.camera.PersistType.All)
            cam.set_pixel_format(vimba.camera.PixelFormat.Bgr8)

            frame_queue = queue.Queue()
            recorder = ImageRecorder(cam=cam, frame_queue=frame_queue)

            # Start a thread that runs write_image(frame_queue). Marking it as daemon allows the python
            # program to exit even though that thread is still running. The thread will then be stopped
            # when the program exits

            threading.Thread(target=functools.partial(write_image, frame_queue, path, pic_format), daemon=True).start()

            recorder.record_images(num_pics=num_pics)
            frame_queue.join()

capture_picture('test', 'jpg')

I had to change the sleep time from 0.1 to 0.03 to get anywhere near the desired number of images. For example, if I used 0.1 sec while trying to capture 30 images, over 90 images were saved instead.

However, with all attempts I always have the problem that, for example, with 30 images, the first maybe 10-15 are saved correctly, while the following images are already captured images.

newtop95 commented 3 years ago

I tested if i have issues with the queue, bit i can't find an error.

I tested the number of frames put in the queue....

def __call__(self, cam: vimba.Camera, frame: vimba.Frame):
        # Place the image data as an opencv image and the frame ID into the queue
        #sanity check for complete frames
        if frame.get_status() == vimba.frame.FrameStatus.Complete:

            self._queue.put((frame.as_opencv_image(), frame.get_id()))
            print('put %s in the queue' %(frame.get_id()))
        # Hand used frame back to Vimba so it can store the next image in this memory
        cam.queue_frame(frame)

and also the popped of frames ...

def write_image(frame_queue: queue.Queue, path, pic_format):

    while True:
        # Get an element from the queue.
        path = path
        frame, id = frame_queue.get()
        print(f'popped frame {id} of the queue')
        cv2.imwrite(f'/home/pi/Desktop/pictures/%s/{id}.%s' %(path,  pic_format), frame) #datetime.now().timestamp(), pic_format), frame)
        # let the queue know we are finished with this element so the main thread can figure out
        # when the queue is finished completely
        frame_queue.task_done()

The output was just fine:

put 0 in the queue
popped frame 0 of the queue
put 1 in the queue
put 2 in the queue
popped frame 1 of the queue
put 3 in the queue
put 4 in the queue
put 5 in the queue
popped frame 2 of the queue
put 6 in the queue
put 7 in the queue
put 8 in the queue
popped frame 3 of the queue
put 9 in the queue
put 10 in the queue
popped frame 4 of the queue
put 11 in the queue
put 12 in the queue
put 13 in the queue
put 14 in the queue
popped frame 5 of the queue
put 15 in the queue
put 16 in the queue
put 17 in the queue
popped frame 6 of the queue
put 18 in the queue
put 19 in the queue
put 20 in the queue
popped frame 7 of the queue
put 21 in the queue
put 22 in the queue
put 23 in the queue
put 24 in the queue
popped frame 8 of the queue
put 25 in the queue
put 26 in the queue
put 27 in the queue
popped frame 9 of the queue
put 28 in the queue
put 29 in the queue
put 30 in the queue
popped frame 10 of the queue
put 31 in the queue
put 32 in the queue
put 33 in the queue
put 34 in the queue
popped frame 11 of the queue
put 35 in the queue
put 36 in the queue
put 37 in the queue
popped frame 12 of the queue
popped frame 13 of the queue
popped frame 14 of the queue
popped frame 15 of the queue
popped frame 16 of the queue
popped frame 17 of the queue
popped frame 18 of the queue
popped frame 19 of the queue
popped frame 20 of the queue
popped frame 21 of the queue
popped frame 22 of the queue
popped frame 23 of the queue
popped frame 24 of the queue
popped frame 25 of the queue
popped frame 26 of the queue
popped frame 27 of the queue
popped frame 28 of the queue
popped frame 29 of the queue
popped frame 30 of the queue
popped frame 31 of the queue
popped frame 32 of the queue
popped frame 33 of the queue
popped frame 34 of the queue
popped frame 35 of the queue
popped frame 36 of the queue
popped frame 37 of the queue
NiklasKroeger-AlliedVision commented 3 years ago

I cannot see anything wrong with the code from reading through it right now in regards to the same image being saved multiple times. I am however slightly concerned, that you are seeing more than 30 frames being added to the frame queue since you are only executing 30 software triggers. The camera should, if set up correctly, only record exactly these 30 images. Also the delay between executions of the software trigger should not have any impact on the number of frames being recorded, if the camera is set up to record images on software triggers.

I see that you are no longer using the _setup_software_tiggering method I have used in my initial code example. You could try adding that back and using it to ensure that the camera is using the software trigger as expected (if you want to use software triggering at all).

However, with all attempts I always have the problem that, for example, with 30 images, the first maybe 10-15 are saved correctly, while the following images are already captured images.

This sounds like there is still something odd going on with the frames being not transferred correctly.

Since the frame buffers are reused in asynchronous image acquisition, the memory into which an image is written upon transfer might have already been used by a previous image. Something like this might be going on (though the FrameStatus test you are performing should prevent this!).

At this point if for some reason the frame does not indicate that it is incomplete (or the check is not performed at all), the same pixel data is encountered in the callback as for Frame 1. That is the only way I can image how you end up with the same image multiple times.

If Vimba does not set the frame status correctly, that is a serious problem as this is something you should be able to rely on! But I have never encountered a problem where an actual incomplete transfer is not marked as incomplete... So I am afraid I will need to do some more digging before I can provide an actual answer. I will try to replicate your problem with the code you have given. Would i be possible for you to also provide the camera settings XML file?

newtop95 commented 3 years ago

@NiklasKroeger-AlliedVision With the software trigger reinstated , it runs.... i don't know what i did there i might have deleted it by mistake. Sorry for wasting your time, but thank you for your support.