Multiprocessing and multiple cameras?

[FlorentLM]

Hello,

I have five acA1440-220um cameras synchronised by an external hardware trigger (an Arduino).

I can successfully retrieve frames data at 215-220 fps at the cameras native resolution (1440x1080) with no overtriggers events.

Looking at the usbtop command output, all my 5 cameras transfer data at a stable 330 000 kbps rate, so far so good.

I use numpy arrays as frames buffers, with two alternating buffers per camera: once the first is full, they're swapped and the second starts to fill, then they swap again, etc. For instance with buffers of length, say, 100, they are swapped every 100 frames and this works well. Still good.

Now I would like to save to the disk the buffer that is not written into, as soon as it's available (i.e as soon as the buffers swap).

I tried implementing this with two threads per camera (one 'grabbing' thread and one 'saving' thread), and with 5+1 threads (one 'saving' thread per camera and one thread to grab from all of them using a pylon camera array). If I use less than 5 cameras, both approaches work well, all frames are saved at 215 fps and usbtop says 330 000 kbps for all cameras. Good.

However, when I have the 5 cameras connected, things get weird: if I start more than 5 threads, the framerate drops dramatically, usbtop says 20 000 kbps per camera. This is actually the case even if the 'saving' threads don't actually do anything (commenting out the bit that actually saves data).

Could this be the result of the GIL?

Is there any way around this? I couldn't manage to make use of multiprocessing with pypylon... I'm thinking 1 process per camera with two threads. Would that be achieveable?

Or am I forced to write everything in C/C++/C#?

Cheers!

[SMA2016a]

yes, this could be an issue of GIL. But I am also not sure. This can also an issue of hard drive. Log the camera's streaming parameters to see if buffer underrun is occuring?

And also the time that is consumed to saving image. May it will randomly very high due to parallel activity of hard drive.

[weertman]

Wow! Sounds amazing @FlorentLM could you share the code? getting multiple cameras to work quickly is a pain in pypylon.. almost like the devs want us to buy software..

[thiesmoeller]

Hi @weertman,

can you describe your multi camera application scenario? There are different ways to run pypylon with multiple cameras. Some of them are described in companion repository https://github.com/basler/pypylon-samples

To use pypylon with multiprocessing is not really a pypylon specific topic, as pypylon's output format are standard numpy ndarrays.

Pushing large amount of data between processes on the same host is a topic of its own. If you have solved this ( there are numpy examples that demonstrate exactly this ), then plugging pypylon into the game is a tiny step.

So if you can describe your scenario, it might be a good starting point to commonly work on a programming sample for this project

Thies

[thiesmoeller]

... just a quickly hacked together example uses https://github.com/portugueslab/arrayqueues. might not be optimal, but is an example to overcome the GIL by using multiprocessing and shared memory

from arrayqueues.shared_arrays import ArrayQueue, ArrayView
from multiprocessing import Process
import numpy as np
import pypylon.pylon as py

class MetaArrayQueue(ArrayQueue):
    """A small extension to support metadata saved alongside arrays"""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def put(self, element, meta_data=None):

        if self.view is None or not self.view.fits(element):
            self.view = ArrayView(
                self.array.get_obj(), self.maxbytes, element.dtype, element.shape
            )
        else:
            self.check_full()

        qitem = self.view.push(element)

        self.queue.put((meta_data, qitem))

    def get(self, **kwargs):
        meta_data, aritem = self.queue.get(**kwargs)
        if self.view is None or not self.view.fits(aritem):
            self.view = ArrayView(self.array.get_obj(), self.maxbytes, *aritem)
        self.read_queue.put(aritem[2])
        return meta_data, self.view.pop(aritem[2])

class ImageProcess(Process):
    def __init__(self, source_queue):
        super().__init__()
        self.source_queue = source_queue

    def run(self):
        while True:
            meta, img = self.source_queue.get()
            print(np.mean(img), meta)

class CameraProcess(Process):
    def __init__(self, source_queue):
        super().__init__()
        self.source_queue = source_queue
        self.tlf = py.TlFactory.GetInstance()

    def run(self):
        cam = py.InstantCamera(self.tlf.CreateFirstDevice())
        cam.Open()

        cam.StartGrabbing()
        while True:
            with cam.RetrieveResult(1000) as res:
                if res.GrabSucceeded():
                    # one copy to get the frame into python memory
                    # this would be possible to skip, but then the release of the buffer
                    # would have to be synced with 'ImageProcess' beeing finished with processing of frame
                    img = res.Array
                    self.source_queue.put(img, {"timestamp": res.GetTimeStamp()})

if __name__ == "__main__":
    q = MetaArrayQueue(256)  # intitialises a MetaArrayQueue which can hold 256MB of data
    i_proc = ImageProcess(q)
    c_proc = CameraProcess(q)
    # start both processes
    i_proc.start()
    c_proc.start()
    # wait for completion ( in this example never )
    i_proc.join()
    c_proc.join()

[weertman]

Hi @weertman,

can you describe your multi camera application scenario? There are different ways to run pypylon with multiple cameras. Some of them are described in companion repository https://github.com/basler/pypylon-samples

To use pypylon with multiprocessing is not really a pypylon specific topic, as pypylon's output format are standard numpy ndarrays.

Pushing large amount of data between processes on the same host is a topic of its own. If you have solved this ( there are numpy examples that demonstrate exactly this ), then plugging pypylon into the game is a tiny step.

So if you can describe your scenario, it might be a good starting point to commonly work on a programming sample for this project

Thies

Hey Thies!

This is great!

Thank you for the question. I come from the world of behavioral neuroscience, so watching animals do stuff, often with devices in them.

In this field of work, 3-D pose estimation has become a really powerful tool. If your interested in some details of what is possible check out this recent review on the topic Leaving Flatland: Advances in 3D behavioral measurement. What is possible with the technology is quire remarkable.

The requirements for 3D-Pose to work well are almost a worst case scenario for the cameras. You want a good framerate in the range of 10-90 fps (scene/species dependent), the cameras need to be time synchronized <~10 ms (around 1 ms is desired), and the recordings often need to be long, > 1 hr (compression can be done post).

And finally, the real compute problem. More cameras is better. 3 is better than 2, 4 is better than 3, all the way up to around 6-12ish cameras (scene/species dependent) when you start to get diminishing returns. The only nice part of it all is that you don't need high resolution cameras. Required resolution (scene/species dependent) can be as low at 600x400.

Basically, as many cameras as possible, as fast as a computer can go, recording as long as possible, with hardware sync. Assume the computer is a beast (all the CPU, all the RAM, all the ports).

I'm ok at python, I'm a data analyst/grad student who migrated from R so once I delve into using classes I get kind of confused. I found trying to get multiple Basler cameras to run on a system in parallel quite confusing from scratch. At least when I was writing my scripts (~1 yr ago) there were no good multi-camera example scripts for Pypylon. There was a multi-camera example I got working and built a fairly sophisticated script around, but it was all on a single processor so 3-D pose was impossible, and framerate was limited. We've started using some fairly old but functional expensive pay to use software (StreamPix9) to deal with this problem.

Cheers! Willem

[thiesmoeller]

moved the discussion into https://github.com/basler/pypylon/issues/569

[FlorentLM]

@weertman Sorry for the delayed reply! Turns out we work on the same thing, I am also in behavioural neuroscience :) I just uploaded a (more or less) clean version of my code, you can check it out at https://github.com/FlorentLM/mokap

Sorry it's completely lacking in terms of comments and such, but I'll work on that asap.

[weertman]

@FlorentLM, sorry for slow response, this is great thank you! It looks like in hardware.py there are some user defined serial numbers. These are what I should change for my own cameras? readme == TODO :)