Native Mobile Applications - Wrap PWA - Githubissues

sign / translate

Effortless Real-Time Sign Language Translation

https://sign.mt

Other

446 stars 77 forks source link

Native Mobile Applications - Wrap PWA #25

Open AmitMY opened 2 years ago

AmitMY commented 2 years ago

Expected Behavior

Allow installing apps through the app store. Models should work fast on mobile as well.

Actual Behavior

App is a PWA, installation only via the browser
Models (holistic, pix2pix) are slow.

Suggested steps

[x] Use capacitor to create native binaries for iOS and Android
[ ] Write a MediaPipe plugin to run Holistic natively
[ ] Write a tflite plugin to allow running other models, like pix2pix
For iOS, use CoreML models
- [x] Convert models to MLPackage
- [ ] Perform inference using CoreML

AmitMY commented 2 years ago

Native build bugs:

[x] Speech to Text
[x] Share video file on iOS to WhatApp
[x] Extra scrolling on iOS
[ ] Avatar augmented reality not working - https://github.com/google/model-viewer/issues/3548 https://github.com/google/model-viewer/issues/4658
[x] App crashes on "swap direction"
[x] When keyboard opens, hide header