Crop/modify webrender's input

[kenyeresrudolf]

HI Guys, I would really love to create an interactive app, but I have only one camera as an input device. Therefore I would like to make it split screen, by compositing two mediapipe instances, one for the left half and one for the right half of the screen. It is easy to do this with the output, but I would love to do this with the input too. This way the left half of the screen would detect gestures for only the mentioned half of the input. I can transform webrender's output, but it won't change field of view for gesture detection - which I'd love to crop/mask out. Do you have any idea how could i do this? thanks a lot, R

[domisjustanumber]

Hi @kenyeresrudolf that sounds super cool!

Right now the best (if not the most elegant) way to do this would be as follows:

Video Device In > Syphon Spout Out > OBS Studio > VirtualCam plugin

The VirtualCam plugin gives you multiple virtual webcams (rather than just the 1 by default), so you can set it up so it sends the same Spout video source out to 2x Virtual Webcams.

You can then run 2x instances of the MediaPipe plugin, with each one using a different virtual webcam as it's media source. You could either crop the video in TD before sending it out via Spout, or in OBS before sending it to a virtual webcam.

It means you have to run OBS in the background the whole time, but until we can figure out a more convenient way to get TOPs into the web browser that MediaPipe runs in, this will work for now

[kenyeresrudolf]

hi @domisjustanumber , Thank you for the quick reply.

It seems that the OBS double (virtual) output method could work, it does not require sending the webcam picture from TD to Spout either, as it is possible to cut/mask the camera image internally in OBS.

You are correct, it means that the OBS will eat some computational power. It would be a way more convenient way to modify the TOPs that feed the web browser internally in TD. If you can add this feature to the update, that would be greatly appreciated.

This plugin is already a gamechanger, thank you a lot for your hard work!

[kenyeresrudolf]

hi @domisjustanumber ,

Sorry, I tried to build this up in my project, but the method seems not working, as I could not create two half webscam screen input to feed the mediapipe instances:

• following your guidance: from TD i can send two separate outputs with syphonspoutout, but in OBS I will see only one
• while in OBS if I create two instances for a videocapture device with copypasting, then i cannot apply different crop settings to them
• if I add a new videocapture device, then I could add a new filter with different crop settings. BUT if I am using the the same webcam as a source, then only one of those will run. Maybe it is a bug in OBS itself. Please let me know if you succeed, maybe i forgot something or in case if you have any other idea :)

thanks a lot

[domisjustanumber]

Ah you're so close! Instead of using the VirtualCam under Tools, you create a scene for each crop you want, then apply a VirtualCam filter to each scene that sends each scene out to a different VirtualCam

Source Filter section of https://obsproject.com/forum/resources/obs-virtualcam.1744/

[domisjustanumber]

I had another thought about this - I also have a webcam I can only run once, but I'm able to get this solution to work and I think the answer is not using source filters.

If you create 2 scenes in OBS and add the same webcam source to each, you can then crop the video n each scene using Alt + drag. That lets you have the same source in as many places as you want without issue.

Once you've set your scenes up, you then add a virtualcam filter to each scene to send it to a different virtual camera.

[kenyeresrudolf]

hi @domisjustanumber, Yes, it is working now. My only issue here, is that it looks like mediapipe stretch the analysed image to the 16:9 ratio, despite it is cropped to 8:9. It recognizes the hand gestures, but the overlay is shifted and stretched therefore I assume this workaround result in some kind of performance drop. Did you experience the same? Again, it is working, so I am really appreciating your help! But in the long run, I believe this could be improved!
Thanks again!

[domisjustanumber]

I made a quick tutorial video on how to set it all up so you get a non-stretched version for each side of the screen - hopefully it's what you're trying to achieve

https://youtu.be/9OFfWGsBIyM

[kenyeresrudolf]

Hey Dom,

Thank you a lot, almost there! I still have some issues on maintaining the performance but on the way! Thank you a lot for your kind help!

[kenyeresrudolf]

Hi @domisjustanumber Thanks again, i made it till some kind of mvp, please find a quick mood video about it here. Hope that i will have some time and bring it to next level in the couple weeks. https://youtu.be/BnYx9W8dXKI The main learning to me about MP that it is really dependent on the light conditions, gpu/cpu load, sensitive to fast fast movements and some people just have different hands than others in the learning base :) thanks again, please continue your journey as this thing is really cool!

[domisjustanumber]

Im so glad you managed to get it working and thanks so much for letting me know what you've been making! That's such a cool video, I love the mood!

And yeah, the lighting and distance is definitely a thing. I was using the plugin for an event this weekend and everything worked great until we turned on the very red lighting for the night and suddenly all my hand detection stopped working. I managed to get some tweaks to the colours and lighting and got it all working in the end though

[kenyeresrudolf]

Thank you! Do you think it something that will be changed with the new releases? Did you do the tweaks with the lights or internally in TD? I tried to feed the webcam in bw for example, but experienced somehow worsened performance

[domisjustanumber]

All of the actual machine vision parts of the library are all Google, so not something we have any control over. They are cranking out monthly updates to the library though, so it's possible the tracking models will get updated at some point.

The source images the model was trained on are in colour, so having your video in natural looking colours too definitely makes the tracking better.

As for tweaking the colours, the white balance, exposure, saturation and maybe other settings on the webcam itself can be useful. You can usually access them by clicking the Device button on the Video Device In TOP. Once you're in TD HSV Adjust TOPs can be useful as can any other TOP that lets you adjust colours. Run the video through whatever you need there to make it look better, then route it through SpoutCam or OBS to the MediaPipe plugin