search

EddyVerbruggen / nativescript-plugin-firebase

:fire: NativeScript plugin for Firebase
https://firebase.google.com
MIT License
1.01k stars 446 forks source link

Add ML Kit support #699

Closed EddyVerbruggen closed 6 years ago

EddyVerbruggen commented 6 years ago

Yesterday at Google I/O a new feature was announced for Firebase: ML Kit. Let's see how neatly we can integrate its features into this plugin.

Feature checklist

If you're curious about the implementation: I'm continuously pushing my changes to this branch.

The checkboxes indicate my progress (on both platforms). Some items may move to different issues/versions.

Generic

Text Recognition

Barcode Scanning

Face Detection 👩‍👧‍👦

Image Labeling 🏷 🏷

Landmark Recognition 🗼🗽

Custom Models (moved to 6.1.0, see #702)

~Use a custom TensorFlow Lite model, on the device~ ~Use a custom TensorFlow Lite model, stored in Firebase (so users always have the latest model)~

jlooper commented 6 years ago

Yasssss!!!! So excited by this!

davidpanic commented 6 years ago

Is there some sort of beta on npm yet? I'd really like to use this in my app, or at least try it out. If there's no beta, can you reference some build instructions please, I wasnt able to find any :stuck_out_tongue_closed_eyes: EDIT: nevermind I found the pack.sh script under publish.

EddyVerbruggen commented 6 years ago

@psrcek Thanks for asking! As you found out you can create a local package of this branch, but you're best off waiting a day or two because I'm currently hammering down on this feature like a madman. Expect it to be more stable very soon!

davidpanic commented 6 years ago

So what you're saying is that you will make a release after a day or two with mlkit included?

EddyVerbruggen commented 6 years ago

My comment was about trying this branch.

I'll make a release once I'm happy with all parts of the release and there's no regression. That's not gonna happen in two days, but I hope to find enough time to release it next week.

ZachRoberts25 commented 6 years ago

@EddyVerbruggen Hey dude, you are the man. Thanks for the great work! Following :)

davidpanic commented 6 years ago

@EddyVerbruggen I've successfuly used the new features! Seems awesome, the only thing i'm bummed about is that i can't get the elements and their location in the text recognition like I can in pure java on android: image

By looking at the code it doesn't seem that hard to add and it is IMO a must have feature!

EddyVerbruggen commented 6 years ago

Ha, @psrcek very cool that you've already tried it! Indeed, that feature is not exposed through the plugin yet but it can easily be added. I'll add it to the featurelist above.

To anyone interested, I just published a beta release: tns plugin add nativescript-plugin-firebase@next. It contains most features I planned in the original post above. Note that there's no documentation yet, but there's a very elaborate demo app that should help you get started.

@psrcek In your case you seem to know how to install the plugin from a cloned Git repo so don't bother installing the beta.

EddyVerbruggen commented 6 years ago

@psrcek The ML Kit branch has been updated to give you more details when recognizing text locally on a device.

If you're using a still image then it's good to know the function name has changed from recognizeText to recognizeTextOnDevice because there's now also recognizeTextCloud (and the input and output params of those functions is different). The Cloud bits of the FIrebase SDK are in 'preview' mode and will probably change. Text recognition is probably more accurate but because of the unstable API in their SDK I simply return the raw text back to the client (for now).

EddyVerbruggen commented 6 years ago

UPDATE:

jlooper commented 6 years ago

Over-the-moon excited to try this, just in time for My Epic Next Demo 💃 #ThanksEddy

davidpanic commented 6 years ago

@EddyVerbruggen I used the new text recognition and I have a suggestion: your interfaces should look like this IMO to reduce confusion, somewhat match the java implementation and make lines accessible (they aren't right now):

export interface MLKitRecognizeTextResultBounds {
    x: number;
    y: number;
    width: number;
    height: number;
}

export interface MLKitRecognizeTextResultElement {
    text: string;
    bounds: MLKitRecognizeTextResultBounds;
}

export interface MLKitRecognizeTextResultLine {
    text: string;
    bounds: MLKitRecognizeTextResultBounds;
    elements: MLKitRecognizeTextResultElement[];
}

export interface MLKitRecognizeTextResultBlock {
    text: string;
    bounds: MLKitRecognizeTextResultBounds;
    lines: MLKitRecognizeTextResultLine[];
}

export interface MLKitRecognizeTextOnDeviceResult extends MLKitResult {
    blocks: MLKitRecognizeTextResultBlock[];
}
davidpanic commented 6 years ago

I made a pull request with changes to the android code, I didn't change the iOS code though as I don't have a mac so I haven't even tried making anything with iOS and I don't know how to code for iOS. :rofl:

EddyVerbruggen commented 6 years ago

Hey @psrcek I'll check it out.. I was actually implementing your changes myself and have the iOS part ready so should be easy :) Thanks!

UPDATE: yep, we aligned quite perfectly. The only change I'm making is the type of MLKitRecognizeTextResultBounds because of alignment with iOS (and IMO it's clearer).

export interface MLKitRecognizeTextResultBounds {
    x: number;
    y: number;
    width: number;
    height: number;
}

Becomes

export interface MLKitRecognizeTextResultBounds {
  origin: {
    x: number;
    y: number;
  },
  size: {
    width: number;
    height: number;
  }
}
EddyVerbruggen commented 6 years ago

6.0.1 has been RELEASED!

To configure and play with ML Kit, follow the instructions in the doc.

In case of issues: please post them here, on GitHub. Not on Slack/Twitter/Email. Thanks!

davidpanic commented 6 years ago

@EddyVerbruggen I made my friend lend me his mac and tried to compile my project on iOS, but I keep getting this message spammed in console whenever i try to use any live preview components:

An empty result returned from from GMVDetector for VisionTextDetector.

The preview is displayed just fine, but does not detect anything. (keep in mind that my project works fine on android)

EDIT: I found this on StackOverflow, it seems like it's a bug.

EddyVerbruggen commented 6 years ago

@psrcek Is that with the demo app mentioned at the top of the ML Kit readme in this repo (linked in my previous post)? If not, please try that first. The steps are in the readme. It's working fine on my iPhone X...

Otherwise please share a repo reproducing this issue. For instance, do you have all these set to true?

The reason I ask is because the message "An empty result returned from from GMVDetector for VisionTextDetector." is a symptom of the GoogleMVTextDetectorResources.bundle not being part of your project.

Btw please open a new issue as this one is closed and shipped ;)

davidpanic commented 6 years ago

Yeah, oops i only had "ml_kit" set to true, you must have added those at some point in between me trying this :stuck_out_tongue: And i'm sorry about not opening another issue, i'm a derp.

EddyVerbruggen commented 6 years ago

That's a very good point, I did add those (I don't want to ship stuff with your app you don't actually need). 👍

coderaven commented 5 years ago

I just want to leave a comment here that you all sirs are heroes to us! Thanks for making this plugin happen! 😄