search

fabio914 / RealityMixer

Mixed Reality app for iOS
GNU General Public License v2.0
776 stars 63 forks source link

Add support Android 9 or higher #59

Open Githubmaster9089 opened 3 years ago

Githubmaster9089 commented 3 years ago

Dear fabio914. I am wondering if u are also developing an android version of that application.

Regards, Githubmaster9089

ilker-aktuna commented 3 years ago

yes. we all wonder that. and I am ready to pay for it :)

Githubmaster9089 commented 3 years ago

I am ready to pay money for sth. like that too

ilker-aktuna commented 3 years ago

actually I am able to develop android apps but I know nothing about image processing. If I had a clue on where to start I would be interested in developing this for Android.

fabio914 commented 3 years ago

@ilker-aktuna You might want to start by taking a look at the Oculus MRC plugin for OBS and checking how the Mixed Reality video is formatted and how it's sent over the network. Then you'll want to check how to integrate FFMPEG into your Android app and how to decode the video and present it in a View. After that you can start checking how to obtain the video from the camera, and how you can compose the Mixed Reality scene (by placing the video from the camera in between the foreground and background videos). Unfortunately, there's no simple option for the "person segmentation" on Android. You can try to adapt an ML model that's typically used in a desktop application (example), but unless it's done properly and the phone has specific hardware that can run this model faster, the results will likely be very slow. You can instead play around with some chroma key shaders, to replace the color green with transparency, and apply those on top of the image from the camera, but then the user will have to use a green-screen.

zhubinsheng commented 2 years ago

https://github.com/ZHKKKe/MODNet run slow ,only 2or3 fps on android

ilker-aktuna commented 2 years ago

@zhubinsheng what is this ? seems not even related to the topic. what are you trying to tell ?

zhubinsheng commented 2 years ago

@zhubinsheng 这是什么？ 看起来甚至与主题无关。 你想告诉什么？

I want to build an android app that is similar to your idea. Using AI model to run is too slow

ilker-aktuna commented 2 years ago

ok. I also "want" to build. But I don't have the courage to start from zero. So I am unable to build right now. But what's the reason to share something not really very related to the topic ? If there is any relation, can you explain ?

PaulMndn commented 9 months ago

@fabio914 It looks like Google has released mlkit which includes a "person segmenter", they call it "Selfie segmentation", but as I understand it, it works with any image or video, including live video. With that it should be possible to create a port for android, right? They say the API has a varying latency depending on the image resolution, but I imagine that would be quite stable for this application, and the app could just include a slider to adjust the offset of the two video sources.

Would it be possible to port this app to android using this feature? (also, possibly @ilker-aktuna)

happyplacetraveler commented 9 months ago

Heck, can you port it back to ios and maybe to mac for fun. : m)  Not having something to mix reality stinks! I miss it. Sent from my iPhoneOn Oct 5, 2023, at 5:40 PM, PaulMndn @.***> wrote: @fabio914 It looks like Google has released mlkit which includes a "person segmenter", they call it "Selfie segmentation", but as I understand it, it works with any image or video, including live video. With that it should be possible to create a port for android, right? They say the API has a varying latency depending on the image resolution, but I imagine that would be quite stable for this application, and the app could just include a slider to adjust the offset of the two video sources. Would it be possible to port this app to android using this feature? (also, possibly @ilker-aktuna)

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

fabio914 commented 9 months ago

@PaulMndn something like might work in an Android app for Mixed Reality Capture. The issue today is that Meta stopped supporting Mixed Reality Capture (the official Mixed Reality Capture Calibration app is broken now that they've changed the Android version that they're using), and many games no longer support Mixed Reality Capture (for example, I tested Beat Saber and Synth Riders recently, and they also stopped working).