search

torinmb / mediapipe-touchdesigner

GPU Accelerated MediaPipe Plugin for TouchDesigner
MIT License
520 stars 21 forks source link

RTSP Feed instead of using Webcam? #76

Open Ganesh-Baron-Aloir opened 2 months ago

Ganesh-Baron-Aloir commented 2 months ago

Hello Torin,

Awesome work on this repo. I'm trying to figure out how it would be possible to modify the code to enable RTSP feed input instead of a webcam.

The reason, I'm trying to figure this out is because, I realize that when needing multiple types of data, faces, landmarks, and so on, we get better results by running multiple MediaPipe bases for marginal performance costs. I'm using NDI webcams instead of Spout right now, because it allows me to share the same feed over 4 MediaPipe bases on the same machine, but I'm limited by the 4 NDI Cam sources.

I believe that with a multicast RTSP it would be possible to use the same source across multiple instances of MediaPipe bases.

domisjustanumber commented 2 months ago

Hey @Ganesh-Baron-Aloir glad you're having fun with the plugin! We looked into the RTSP option, but it's a bit of a pain to get working, and requires you to have the licensed version of TouchDesigner to run. I haven't played with it to test, but I also suspect the latency might be quite high.

I'm currently (well, I will be when I next get some time to tinker) looking at capturing a window as the video source, which would (I believe) allow us to have unlimited number of MediaPipe instances all working from the same source material with 0 delay... you can keep track of it here: https://github.com/torinmb/mediapipe-touchdesigner/issues/64

Ganesh-Baron-Aloir commented 2 months ago

Hey @domisjustanumber I can understand the challenge of integrating RTSP. Good point on the latency, that is indeed a possiblity.

If I get some time I might look at this external python library: pyvirtualcam it might allow to get the data from a TOP of some GLSL wizardry and virtualize a camera.

domisjustanumber commented 3 weeks ago

Thanks for the kink, thats an interesting tool! Dotsimulate has also done some cool stuff with pulling GPU memory directly into Pyhton, but alas MediaPipe is Javascript running in embedded Chromium. As it's a web browesr, it's very heavily walled off from memory and network access.