[HOW-TO] grab frames or arrays from stream

[KruemelmonsterK]

My goal is to do multi object tracking (MOT).

In my head, the steps to do so are:

1. capture frame with camera NoIr v3

2. transfer frame from the Raspberry Pi to a stronger machine (currently my laptop)

3. On the stronger machine (laptop): run algorithm for MOT (e.g. YOLO) on that frame

4. Repeat from step 1 with the next frame

Currently I am able to to do videos in format mjpeg with size 1980 x 1020 at 30 fps.

My question is: how can I hand the frames (wihout loss of frames) from the Raspberry Pi over to my stronger machine via e.g. Ethernet?

[KruemelmonsterK]

I decided to use socket to send every array from the raspberry pi to my stronger machine.

On the raspberry pi I will create an array with: frame = picam2.capture_array()

Then I'll have to send this array to my stronger machine with socket. There I will process every frame with YOLO and afterwards display the image with: cv2.imshow("picam2", frame)

Since I dont have a lot knowledge about socket, I wonder if it is possible the simply get the frames from a stream. The disadvantage I see with this is, that I cant stream raw files and so, I'll have to transform the raw files first to normal image. The normal image then will be send to the stronger machine, where i'll have to transform it to an array again. This seems to be too computationally intensive for real-time.

[davidplowman]

Perhaps you could encode the video stream and forward it to the socket? There are examples of this here where the stream is encoded using the h.264 encoder to reduce the amount of data. You could use the MJPEGEncoder instead for an MJPEG stream.

In both cases you'd have to decode the images on the receiving end. If you used Encoder (instead of H264Encoder) then you'd get uncompressed images, though you'd have to know exactly how big they are and in what format, to understand them!