search

techyian / MMALSharp

C# wrapper to Broadcom's MMAL with an API to the Raspberry Pi camera.
MIT License
195 stars 33 forks source link

Question: LineDetection in Video #199

Open ReinhardGruber opened 3 years ago

ReinhardGruber commented 3 years ago

Is it possible to use the sobel line detection in video mode and display in preview?

MV10 commented 3 years ago

I don't think that's possible at the moment. Effects and similar processing operates on the output of the hardware pipeline.

Last year I was trying to figure out how to do motion detection earlier in the pipeline (at the port level) and concluded it wasn't possible, but Ian speculated that it might be possible with changes. I'll take his word for it, he's the MMAL ninja around here. But right now only built-in features like the resizer work in the middle of the pipeline.

I was barely able to get video of the motion detection processor to work by piping the raw output through ffmpeg and clvc to an MJPEG stream (v0.7 wiki), but that's low resolution (640x480 is optimal) and even at low res the matrix effects run too slowly for real time video -- depending on matrix size you're doing 9X or even 25X the number of passes per pixel that motion detection is doing.

ReinhardGruber commented 3 years ago

ok. is it possible to modify the preview stream frame by frame? (like the resizer)

Framerate: 1 frame per second is enough

MV10 commented 3 years ago

No, that's what I was trying to explain -- image effects only operate on pipeline output, there's currently no way to feed any sort of complex (frame-level) processing back into the pipeline. The preview stream is part of the pipeline. With the current design, preview output can only be altered by built-in features (like the resizer).

(I say "complex" processing because there are places within the pipeline where very simple processing could alter the image, but the amount of data available can vary considerably and unpredictably, so I think only individual pixel-level effects are reasonable with the current design. Color filters, for example. Line detection is very complex, relatively speaking.)

As for framerate, if I remember right, a 3x3 matrix ran against lower resolutions at about 3 FPS. I would expect .NET 5.0 and later will run much more quickly, although I haven't had time to mess with it.

But unfortunately, I think the bottom line is that the library can't do this yet.