search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://ai.google.dev/edge/mediapipe
Apache License 2.0
27.46k stars 5.15k forks source link

Using Selfie-Segmentation result as a texture on a 3D multiplaying environment (networked-aframe) #2486

Closed jimver04 closed 3 years ago

jimver04 commented 3 years ago

Solution: Selfie-Segmentation.js Language: Javascript Problem: Mediapipe is WASM and has an unexpected behaviour when capturing a video stream from the drawing canvas and passing it with easyrtc to multiplaying experiences. Live demo: https://vrodos-multiplaying.iti.gr/plain_aframe_mediapipe_testbed.html (I have tried Codepen but it is not accepting networked-aframe multiplaying experiences.)

My target is to make a multiplaying 3D environment where the persons are represented by cubes that have as textures their selfie without the background. It is a kind of cheap virtual production without the need of having chroma key background. The background is removed with MediaPipe Selfie-Segmentation. I am using Networked-Aframe with easyrtc adapter as the multiplaying environment.

The full issue description and the code can be found here: https://stackoverflow.com/questions/68994847/using-self-segmentation-javascript-of-mediapipe-to-pass-user-selfie-as-a-texture

The issue is that instead of having the other player texture on the cube (P1 should see P2, and P2 should see P1, each one sees his selfie. This means that P1 sees P1 and P2 sees P2 which is bad.

Tested using 2 devices for each player Desktop: RTX3080 GPU , Windows 10, Intel 10th generation, Chrome, Version 92.0.4515.159 (Official Build) (64-bit) Laptop: RTX3060 GPU , Windows 10, Intel 10th generation, Chrome, Version 92.0.4515.159 (Official Build) (64-bit)

The solution is working well if a draw custom images with canvasCtx.draw, but when using "selfieSegmentation.onResults(drawResults);" It seems that onResults is messing up threads.

mhays-google commented 3 years ago

You can pass a Promise from onResults to prevent execution of the next frame, but we don't have other ways of injecting ourselves into the drawing stream in any good synchronous way at the moment.

google-ml-butler[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

google-ml-butler[bot] commented 3 years ago

Closing as stale. Please reopen if you'd like to work on this further.

google-ml-butler[bot] commented 3 years ago

Are you satisfied with the resolution of your issue? Yes No

jimver04 commented 3 years ago

Actually it was a Networked-Aframe issue that I have solved: https://github.com/networked-aframe/networked-aframe/issues/269