search

MCPCapital / HarmonizeProject

Harmonize Project lets you sync HDMI video with Philips Hue lights using a Raspberry Pi!
231 stars 35 forks source link

[Enhancement] HDMI to CSI adapter #4

Closed MCPCapital closed 4 years ago

MCPCapital commented 4 years ago

Originally opened by @spapadim on August 4th 2020: "Very cool and inspiring project! Always wanted one of these, but off-the-shelf prices are ridiculous..

First thing that immediately came to mind when I saw this: would this work with an HDMI-to-CSI adapter, such as eg https://www.amazon.com/dp/B0899L6ZXZ/ref=cm_sw_r_cp_apa_i_edwkFbJ7V4HT2 ?

Tl;dr: with GPU-accelerated frame processing, I'm hoping this could run even on a Zero..! :)

Other than potentially reducing cost a little bit (CSI adapter vs USB capture), the main advantage is that video downscaling (to compute average color intensities around border) could be offloaded to Pi's GPU (eg, see picamera Python module docs on "advanced configurations" -- GPU can even compute motion vector data quite efficiently, let alone just do pixel averaging). This could possibly allow the use of a smaller Pi, perhaps even a Zero, which would then really reduce cost.

The converters' frame rate is relatively low ("up to" 25fps), but I don't believe this matters here.

The only real catch is that these converters don't do HDCP. But, perhaps, if there is a reasonably priced splitter that can also do HDCP stripping on lower-res port..

I just wish my TV could do full HDMI pass-through (not just ARC to receiver but, eg, for chaining a second monitor), or my receiver could route to both zone's HDMI outputs simultaneously (rather than switching video and splitting just audio), but both of these features seem to be exceedingly rare.. sigh!

Finally, apologies in advance if using an issue for general commenting was inappropriate (feel free to close/delete, of so) -- but just my 2c. I'll probably take this up myself when (if?) I find some time."

MCPCapital commented 4 years ago

Reply by @MCPCapital same day: " @spapadim Thanks for taking the time to write these ideas! I appreciate the time and effort you've put forth. We looked into using one of these adapters initially, but came to the conclusion that a video capture card would be more efficient. We we're testing on a Pi 4B, however, where the CPU is relatively powerful compared to the Zero.

In theory this kind of connection should work as long as the CSI input could be passed through as video0. The python script uses OpenCV to pull video0 to perform its analysis. The splitter in the readme should be apt for this purpose as well. I believe it strips the HDCP out Output 2 (maybe only when downscaling - not sure) so I've never encountered any issues around that.

I don't intend to purchase any more items for this project at this time, but you are more than welcome to give it a try and I'll support however I can. If it is confirmed to work I can add this as an option in the readme as well. Thanks again for your input!"

MCPCapital commented 4 years ago

Reply by @spapadim same day "Thanks for taking time to write reply. I believe utilizing GPU would require switching from opencv to picamera. Camera is indeed /dev/video0 and would work with OpenCV, but without any hardware acceleration.

FWIW, I was playing with a pan-tilt tracker a couple of months ago and there the difference in frame rate is night and day between opencv and picamera. But, in that application, I have MJPEG stream output, and I believe most of speed improvement comes from: (a) offloading JPEG encoding to GPU (needs to be at full res), (b) not having to get a full res frame into opencv anymore (can set up second GPU pipeline to do downsampling etc).

Anyway, if I ever find time to do something, I'll post another comment. Thx again. I'll leave it up to you to close issue."

MCPCapital commented 4 years ago

Reply by @MCPCapital same day: Thinking about it some more, I see what you mean regarding using picamera for GPU utilization, which OpenCV doesn't support. That would require somewhat of a rewrite of the original code, but the logic would be exactly the same. It looks like someone has tried to do this with picamera actually - check this outdated git repo out. Probably doesn't have too much value but from a glance the functions of picamera are pretty similar to cv2.

Also as a note, cv2 uses MJPEG in our implementation, but again I don't think it has the capability of utilizing the GPU. Regarding your (b), I have tried capture cards that further downscale from 1080p to 720p, but those seem to induce more delay than the computational time they save.

I will close this issue for now as I think its outside of the current scope of the project, and a rewrite with new hardware isn't something I have capacity to take on right now. I do, however, appreciate and encourage this discussion! Having an option to use CSI via a simple command line argument or something would be a great enhancement. Feel free to comment and reopen this issue or submit any pull requests as you see fit. Thanks!