search

Qengineering / Multithread-Camera-OpenCV

OpenCV Universal Multi thread video Interface with neglectable latency.
https://qengineering.eu/install-gstreamer-1.18-on-raspberry-pi-4.html
BSD 3-Clause "New" or "Revised" License
22 stars 3 forks source link

Runs only as fast as slowest cam #1

Open Benebaby opened 1 year ago

Benebaby commented 1 year ago

The main loop where all images get stiched together and get previewed runs only as fast as the slowest camera. So if one camera runs on a slow framerate it effects the other cameras wich should not happen.

Qengineering commented 1 year ago

@Benebaby,

You are absolutely right!

To solve it, you have to follow another strategy. The problem is that every application requires a different solution; there is no 'one' genetic outcome.

For instance, you can give each thread its own processing algorithm, leaving the main thread almost empty. There you concentrate on the image stitching.

Or you can signal the main thread when it requests a frame, that is now not ready yet, and it should revisit this thread later again.

The how idea of this repo is a simple starting point for a multithreading camera app.

mxochicale commented 1 week ago

@Benebaby @Qengineering Handling multithreading for cameras at different sampling rates and resolutions presents an exciting engineering challenge. I am eager to spend a few hours at my weekends prototyping these features. It would be fantastic if you could create a branch with an initial approach, and I would be happy to test and contribute to it. Alternatively, a new issue can be raised. For instance, let me share the features of these two cameras (External USB Logitech Webcam C930e and embedded webcam HD Camera: HD Camera) as something to be used to prototype something useful. Thanks, Miguel

See logs for `v4l2-ctl -d /dev/video* --list-formats-ext` ``` $ v4l2-ctl -d /dev/video0 --list-formats-ext ioctl: VIDIOC_ENUM_FMT Type: Video Capture [0]: 'MJPG' (Motion-JPEG, compressed) Size: Discrete 1280x720 Interval: Discrete 0.033s (30.000 fps) Size: Discrete 960x540 Interval: Discrete 0.033s (30.000 fps) Size: Discrete 848x480 Interval: Discrete 0.033s (30.000 fps) Size: Discrete 640x480 Interval: Discrete 0.033s (30.000 fps) Size: Discrete 640x360 Interval: Discrete 0.033s (30.000 fps) [1]: 'YUYV' (YUYV 4:2:2) Size: Discrete 640x480 Interval: Discrete 0.033s (30.000 fps) Size: Discrete 640x360 Interval: Discrete 0.033s (30.000 fps) $ v4l2-ctl -d /dev/video2 --list-formats-ext ioctl: VIDIOC_ENUM_FMT Type: Video Capture [0]: 'YUYV' (YUYV 4:2:2) Size: Discrete 640x480 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 160x120 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 176x144 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 320x180 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 320x240 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 352x288 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 424x240 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 480x270 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 640x360 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 800x448 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 800x600 Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 848x480 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 960x540 Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 1024x576 Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 1280x720 Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 1600x896 Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 1920x1080 Interval: Discrete 0.200s (5.000 fps) Size: Discrete 2304x1296 Interval: Discrete 0.500s (2.000 fps) Size: Discrete 2304x1536 Interval: Discrete 0.500s (2.000 fps) [1]: 'MJPG' (Motion-JPEG, compressed) Size: Discrete 640x480 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 160x120 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 176x144 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 320x180 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 320x240 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 352x288 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 424x240 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 480x270 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 640x360 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 800x448 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 800x600 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 848x480 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 960x540 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 1024x576 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 1280x720 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 1600x896 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Size: Discrete 1920x1080 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) ```
Qengineering commented 1 week ago

@mxochicale, thanks for your enthusiasm! Could you tell me what you have in mind precisely? This repo intends to provide a basic skeleton for multi-threading several cameras (as you undoubtedly understood from the response to @Benebaby). Expanding the functionality with specific cameras, for instance, holds the risk of leaving a generic solution.

mxochicale commented 4 days ago

@Qengineering Agreeing with the goal of having a general-purpose solution rather than a specific one will continuously add significant value to the users of your repository!

For the specific requirement of multiple cameras running at different frame rates, I think a simple example with two cameras would be extremely useful. Once sorted out, it could be scaled to support four or more cameras. For example, I am thinking on the use case with two cameras at different resolutions and different fps with arguments or some kind of configuration file to setup the app. In terms of these two cameras, it could be the case of camera A with 640x480 image resolution, running at 30.000 fps and Camera B with 1920x1080 image resolution, running at 5.000 fps. Similarly, perhaps this would be a good start to think and brainstorm frame lag quantification and synchronisation techniques for threads. What do you think? Perhaps, you would like to create a discussion forum to move such discussion there or create a new issue to not spam @Benebaby and yourself here! Thanks, Miguel

Qengineering commented 4 days ago

@mxochicale, Feel free to fork the repo for add-ons, and extensions. I do not have the time to manage such a project, busy as I am with other (commercial) projects. Good to know, YoloIP and Xactai parking have already incorporated the features you mentioned above!