💭 Question: Implementing Virtual Background and AR Try-On Glasses with Frame Processing

asrasoft-admin commented 3 months ago

Question

Hello,

I am currently developing an application using React Native Vision Camera and I am interested in implementing two specific features using frame processing:

Virtual Background: Similar to the background subtraction and replacement features found in video conferencing apps.
Try-On Glasses: Augmented reality feature to overlay virtual glasses on the user's face.

Questions:

Is it possible to implement real-time virtual background replacement using the frame processing capabilities of React Native Vision Camera? If so, could you provide some guidance or reference on how to achieve this?
For the virtual try-on of glasses, can the frame processor handle augmented reality overlays? Are there any recommended approaches or libraries that integrate well with React Native Vision Camera for this purpose?

What I tried

No response

VisionCamera Version

lts

Additional information

[ ] I am using Expo
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar questions in the issues page as well as in the discussions page and found none.

maintenance-hans[bot] commented 3 months ago

Guten Tag, Hans here.

[!NOTE] New features, bugfixes, updates and other improvements are all handled mostly by @mrousavy in his free time. To support @mrousavy, please consider 💖 sponsoring him on GitHub 💖. Sponsored issues will be prioritized.

mrousavy commented 3 months ago

Is it possible to implement real-time virtual background replacement using the frame processing capabilities of React Native Vision Camera? If so, could you provide some guidance or reference on how to achieve this?

Yes this is possible. We have built this feature for multiple clients in the past already. Previously we created a custom fork of VisionCamera, but nowadays that I built and released Skia Frame Processors it should be possible with just JavaScript.

For the virtual try-on of glasses, can the frame processor handle augmented reality overlays? Are there any recommended approaches or libraries that integrate well with React Native Vision Camera for this purpose?

Sorry I don't understand the question. VisionCamera renders to a 2D surface, so you cannot use it in VisionPro if that's what you're asking.

mrousavy / react-native-vision-camera