Synchronized multi-camera acquisition and recording

[thiesmoeller]

Task is to define a setup and code structure to record synchronized multi-camera data. As the question on how to efficiently handle multi-camera data is a repeating one, final result should be a programming sample.

@weertman provided a real application example to target:

Thank you for the question. I come from the world of behavioral neuroscience, so watching animals do stuff, often with devices in them.

In this field of work, 3-D pose estimation has become a really powerful tool. If your interested in some details of what is possible check out this recent review on the topic Leaving Flatland: Advances in 3D behavioral measurement. What is possible with the technology is quire remarkable.

The requirements for 3D-Pose to work well are almost a worst case scenario for the cameras. You want a good framerate in the range of 10-90 fps (scene/species dependent), the cameras need to be time synchronized <~10 ms (around 1 ms is desired), and the recordings often need to be long, > 1 hr (compression can be done post).

And finally, the real compute problem. More cameras is better. 3 is better than 2, 4 is better than 3, all the way up to around 6-12ish cameras (scene/species dependent) when you start to get diminishing returns. The only nice part of it all is that you don't need high resolution cameras. Required resolution (scene/species dependent) can be as low at 600x400.

Basically, as many cameras as possible, as fast as a computer can go, recording as long as possible, with hardware sync. Assume the computer is a beast (all the CPU, all the RAM, all the ports).

I'm ok at python, I'm a data analyst/grad student who migrated from R so once I delve into using classes I get kind of confused. I found trying to get multiple Basler cameras to run on a system in parallel quite confusing from scratch. At least when I was writing my scripts (~1 yr ago) there were no good multi-camera example scripts for Pypylon. There was a multi-camera example I got working and built a fairly sophisticated script around, but it was all on a single processor so 3-D pose was impossible, and framerate was limited. We've started using some fairly old but functional expensive pay to use software (StreamPix9) to deal with this problem.

Originally posted by @weertman in https://github.com/basler/pypylon/issues/513#issuecomment-1347087048

[thiesmoeller]

Performance requirements:

Resolution: 600x400 AcquisitionFramerate: 90 Num Cameras: 12

@weertman: What is the Pixelformat Mono8/10/12 or one of the color formats, that you use? Do you already have cameras. What is their model name?

[weertman]

Hey @thiesmoller sorry for getting back to you so far out into the future. I just saw this. We have been using Mono8 We have been using a2A2590-60ucPRO not the most practical cameras for our current tracking set up but they were on hand. Basically, we down sample the resolution a bunch before doing deep learning.

[chiyu1203]

Performance requirements:

Resolution: 600x400 AcquisitionFramerate: 90 Num Cameras: 12

@weertman: What is the Pixelformat Mono8/10/12 or one of the color formats, that you use? Do you already have cameras. What is their model name?

Hello @thiesmoeller!! I want to do similar thing but I am still stuck in selecting hardware for the setup. May I ask you for your suggestion on what any of these hardware might be the bottleneck with synchronized multi-camera acquisition and recording?

Resolution: 600x400 Acquisition Frame rate: from 30 to 90 Num Cameras: 6 Camera models: acA1440-220um - Basler ace Mono CPU: AMD Ryzen 5 5600x 6-core Processor (or AMD Ryzen 7 5800x 8-core Processor if needed) RAM: 32GB Disk Write Speed: 1700 MB/s (or 2700 MB/s if needed)

Your feedback would be greatly appreciated! Best Regards Chiyu

[mprib]

@thiesmoeller,

I just stumbled upon this discussion. I maintain a project that performs synchronized recording from multiple webcams for use with markerless tracking:

https://github.com/mprib/pyxy3d

I'm in the process of a fairly comprehensive refactor to de-couple the live camera sync/recording from the calibration process as the initial feedback I've received indicates users are more interested in offline processing of pre-recorded frames (using high resolution/high frame rate devices such as go-pros and iphones).

I just wanted to reach out as I'm hoping to better understand what options are currently out there for these tools and what you may be using (i.e. if there is a good one out there I won't have to maintain my own....which would be nice).

Regards,

Mac