Outdated bindings to Google ML Kit Vision Text Regognition api

[deelw]

Because we’re using ML Kit for Firebase (https://firebase.google.com/docs/ml-kit) for on-device text recognition and this is an old version, we need to migrate to the actual version of ML Kit (https://developers.google.com/ml-kit/vision/text-recognition ).

As Dutch government, we develop our apps using Xamarin and we need to migrate as well our Xamarin.iOS as our Xamarin.Android implementation.

For Xamarin.iOS we’re using nuget package Xamarin.Firebase.iOS.MLKit.Vision version 0.21.0 from Microsoft while for Xamarin.Android we’re using nuget package Xamarin.GooglePlayServices.Vision version 120.1.3, also from Microsoft. These packages contain C# bindings to ML Kit for Firebase.

Surprisingly there seems to be no Xamarin nuget packages available for the current GoogleMLKit/TextRecognition (https://developers.google.com/ml-kit/vision/text-recognition). Neither for Xamarin.iOS nor for Xamarin.Android.

We did try to make bindings ourselves. While we did succeed in making and invoking these bindings, text is never recognized. In the console output the MLKit framework did write the following messages:

• [lvl=3] +[MLKITx_CCTClearcutUploader crashIfNecessary] Multiple instances of CCTClearcutUploader were instantiated. Multiple uploaders function correctly but have an adverse affect on battery performance due to lock contention.
• Failed to locate required resources. Please verify the resource bundle is imported.
• [lvl=3] -[MLKITx_GIPPseudonymousIDStore initializeStorage] Shared App Groups unavailable

We ask you to make nuget packages available for the current versions of Google MLKit, specifically for cocoapod GoogleMLKit/TextRecognition and maven repository com.google.android.gms:play-services-mlkit-text-recognition.

Related issue: https://github.com/xamarin/GooglePlayServicesComponents/issues/420

[deelw]

@SotoiGhost you've helped me a year ago with bindings for iOS. I heard the Components team was dissolved. Could you please help us by either making new Google ML Kit Vision TextRecognition (no Firebase) api's available or support us in making our own bindings work?

As you can read in the issue text, we successfully generated bindings. With our bindings we can invoke the MLKTextRecognizers ProcessImage and ResultsInImage methods. That the framework is successfully invoked is confirmed by the fact that the MLKTextRecognitionCallback of the ProcessImage method is called, but text is never recognized while the error object is null and an instance of the MLKText object is returned. We've tried all orientation flags. In the application console there are three messages (mentioned in the initial issue text above).

Could you or an experienced colleague help us please? Thank you very much in advance!

Here some details for your information:

• I made a podfile (see under) because sharpie pod init 'GoogleMLKit/TextRecognition' resulted in a pod file using pod 'GoogleMLKit'
• I used pod update instead of sharpie because sharpie has no option to use the existing podfile
• after installing cocoapods I needed to call pod repo update to be able to successfully call the pod update command
• Before calling sharpie pod bind I had to unselect Mac for each project target (General tab -> deployment info) in the XCode Pods project (./Pods/Pods.xcodeproj)
• In my Xamarin Bindings library I only included MLKitTextRecognition_ApiDefinitions.cs and MLKitVision_ApiDefinitions.cs and I added a native reference to all the framework files (from the ./Bindings folder).
• In my iOS project I had to set the linker behavior to don't link

Podfile

platform   :ios, '14.3'
--
install! 'cocoapods',   :integrate_targets => false
target   'ObjectiveSharpieIntegration' do
use_frameworks!
 
pod   'GoogleMLKit/TextRecognition'
end

[deelw]

@mattleibow could you please help me with this issue? It's enough for me when I have local working bindings (only text recognition), while it would be very nice when the bindings would be globally available as a nuget package.

[deelw]

Here a solved issue i found with the same error, maybe this helps solving the cause? https://github.com/xamarin/GoogleApisForiOSComponents/issues/382

[mavispuford]

@deelw Were you ever able to get your bindings to work? I'm just starting down that path. Thanks for detailing some of your process in the above comment, by the way.

[deelw]

@mavispuford yes, my bindings did work but without successfull text recognition (the ocr result callback function was called, so the interaction was successfull). However, reproducing the steps now results in a xamarin iOS project with linker errors.

In addition to the above information, I added namespaces to the _ApiDefinitions.cs files and changed the build action to ObjcBindingApiDefinition

Here is my ViewDidLoad implementation with the testing code. Of course I added an bundleresource called identitydocument.png. The object mlkText did have an instance, but the Text property and the Blocks were empty.

    public override void ViewDidLoad()
    {
        base.ViewDidLoad();
        // Perform any additional setup after loading the view, typically from a nib.
        var textRecognizer = MLKTextRecognizer.TextRecognizer();

        var uiImage = UIImage.FromBundle("identitydocument.png");
        if (uiImage == null)
        {
            Console.WriteLine("uiImage IS NULL");
            return;
        }

        var mlkImage = new MLKVisionImage(uiImage);
        if (mlkImage == null)
        {
            Console.WriteLine("mlkImage IS NULL");
            return;
        }

        textRecognizer.ProcessImage(mlkImage, (MLKText mlkText, NSError error) =>
        {
            Console.WriteLine("ProcessImage callback");
            if (mlkText != null)
            {
                Console.WriteLine("mlkText NOT NULL");
                Console.WriteLine($"mlkText.Text {mlkText.Text}");
                Console.WriteLine($"mlkText.Blocks.Length {mlkText.Blocks.Length}");
            }

            if (error != null)
            {
                Console.WriteLine("error NOT NULL");
                Console.WriteLine($"error: {error.DebugDescription}");
            }
        });
        Console.WriteLine("textRecognizer.ProcessImage");
    }

[mavispuford]

@deelw I was able to get iOS bindings working with successful text recognition. My guess is that you were missing the necessary resources in your bindings project that are needed for ML Kit (machine learning models etc.). If you take a look in the MLKitTextRecognition pod folder, you'll see them:

This targets file for the old Firebase ML Kit Vision bindings was a great reference for me, because I later had linking errors in Release mode. Making sure the Frameworks were set up correctly in the bindings project resolved them.

Here's what my csproj file looks like.. The two ApiDefinitions.cs files were generated using the following Sharpie commands:

sharpie bind -o MLKitVisionBindings -sdk iphoneos14.4 -n MLKitVision -framework ./MLKitVision

sharpie bind -o MLKitTextRecognitionBindings -sdk iphoneos14.4 -n MLKitTextRecognition -framework ./MLKitTextRecognition

[deelw]

@mavispuford Thank you very much! I'll try it as soon as possible. I'll let you know

[deelw]

@mavispuford Thank you for your clear explanation. Because sharpie pod bind fails with the latest ios sdk 14.5, I'm trying to do it exactly your way.

For the Dependency/PodBuilder/Prebuilt contents I tried to use PodBuilder like you did. I did make a XCode iOS app project and workspace only because the PodBuilder requires them. Copied my podfile into the project directory. I was able to run pod_builder init. But pod_builder build_all did fail. Can you tell me how you did that?

Thanks for your help!

[mavispuford]

@deelw

Here are the steps that I took for PodBuilder (but note that I ended up dropping it and simplified my bindings project quite a bit - I'll explain below):

• I created an empty XCode project with my podfile referencing the 'GoogleMLKit/TextRecognition' pod
• I ran pod_builder init to create the PodBuilder folder
• Ran pod_builder build_all to build all pods
• The resulting frameworks were located in PodBuilder/Prebuilt
• Pod-Builder skips any pods that already have frameworks. Those pods are in their normal location in the Pods directory.

So I mentioned above that I stopped using PodBuilder to build my fat binaries. The reason is that even though I had success building my bindings DLL and popping it in a simple Xamarin app, it caused problems with our other app because it was referencing Firebase for analytics/push notifications and the two had conflicts with their native dependencies. I looked around and noticed that the majority of the native dependencies are also in Firebase.Core. So in my bindings project, I removed most of the native references and referenced the Firebase.Core NuGet package instead. This works great for us and doesn't have any conflicts with our other packages references anymore.

Here's what the bindings csproj file looks like now.

[deelw]

@mavispuford Yes, it's working! Thanks you very much!

I have one question about your latest solution: How did you prevent the Firebase.Core nuget package from being stripped because it's not explicitly referenced from code? I did so by including a static class with a private static Action reference = Firebase.Core.App.Configure; but maybe you did use a more elegant way?

[mavispuford]

@deelw That's great! I'm glad to hear it.

For Android, we have an r8.cfg to prevent linking with entries like this:

-keep class com.google.firebase.** { *; }
-keep class androidx.camera.core.** { *; }
-keep class com.google.mlkit.vision.** { *; }

As for iOS, I don't think I did anything special to prevent it from being linked out. We were actually already using Firebase.Core in our app, since we use Firebase for push notifications and analytics. Since you're not using it anywhere, you may have to add a Linker.xml entry or just keep that reference around.

[deelw]

Thanks for the info!

[albertruff]

One question: How you defined in MLKitVision_ApiDefinitions.cs the GMLImage_MLKitCompatible Extension/Category, that it extends an Interface?

At the Moment, i create Bindings for the new MLKitBarcodeScanning funktionality. For the MLKitBarcodeScanning.framwork, I successfuly created Bindings. But I stuck At this Point in MLKitVision.

P. S.: Thanks @deelw and @mavispuford for your discussion, it helped me a lot.

[deelw]

@albertruff for MLKitTextRecognition 1.2.0 it works great, but version 1.3.0 I did also not succeed in making working bindings because of the new GMLImage_MLKitCompatible. I tried several ways including modifying the generated bindings. Maybe @mavispuford has a solution? I did take a look at the release notes of MLKit and Google is working on a new version thats in beta MLKitTextRecognition (2) 1.4.0-beta1 So, at this moment you could stay at 1.2.0 or try the beta.

[albertruff]

For the MLKitBarcodeScanning, the 1.4.0 is not in beta, it is the relase version and i use them. For the GMLImage_MLKitCompatible-Problem, I found a working solution but it's not ideal:

• In the MLKitVision ApiDefinition.cs I commented out the interface MLKitCompatible
• Then I build the MLKitVision Project to creat the dll
• After that, I created, for the MLImage, the ApiDefinition.cs and a Binding-Project
• In the MLImage-Binding-Project, I referenced to the MLKitVision.dll
• After that it was possible to implement the IMLKCompatibleImage on the GMLImage

It is the same functionality, that was created in MLKitVision with the MLKitCompatible. The problem is, that in Obj-C it is possible to create an extension for a class to say that this class implements an interface. In C#, I dont found such a possibility.

But with this solution, I have an another problem: When I Build the MLImage, the compiler creates for the following Property a false getter: //@property (readonly, nonatomic) CMSampleBufferRef _Nullable sampleBuffer; [NullAllowed, Export("sampleBuffer")] CMSampleBuffer SampleBuffer { get; }

In the created GMLImage.g.cs the SampleBuffer-Property looks following: [BindingImpl (BindingImplOptions.GeneratedCode | BindingImplOptions.Optimizable)] public virtual global::CoreMedia.CMSampleBuffer? SampleBuffer { [Export ("sampleBuffer")] get { IntPtr ret; if (IsDirectBinding) { ret = global::ApiDefinition.Messaging.IntPtr_objc_msgSend (this.Handle, Selector.GetHandle ("sampleBuffer")); } else { ret = global::ApiDefinition.Messaging.IntPtr_objc_msgSendSuper (this.SuperHandle, Selector.GetHandle("sampleBuffer")); } return ret == IntPtr.Zero ? null! : new global::CoreMedia.CMSampleBuffer (ret); }
}

The problem is the following command: new global::CoreMedia.CMSampleBuffer (ret);. It is not possible to create an CMSampleBuffer with an IntPtr.

@mavispuford Can you Help me? @deelw: Have you seen somthing similar?

[deelw]

@albertruff In C# you can only have extensions for methods. Another way to achieve your goal is to define a class as partial, then you can 'extend' that class with an interface definition by defining a partial class with the same name in the same namespace that extends the interface.

[ElteHupkes]

I'm a bit late to this party (or early depending on how you look at it 😉), but do I understand correctly from the the second comment by @deelw that this repository is essentially no longer maintained?

[deelw]

@ElteHupkes thats right. There are some people maintaining the Android part, but at the iOS side there is no maintenance. Th team was dissolved because everybody had to work on Maui.

[ElteHupkes]

@deelw Well, that's... annoying. A notice on the front page of the repo would've been nice. Preaching to the choir of course, I'm grateful someone knew and shared it here, thanks.

I must say I've avoided the rabbit hole of creating bindings for iOS so far, but if I get to it this time (and succeed) I might turn it into a NuGet package.

[BjornVanslembrouck]

@deelw @mavispuford, I have generated all the binding files I needed and included the machine learning files into my binding project and set them to BundleResource. When I call the processImage(), I don't get any errors, but my text is always an empty string. Did you guys do something specific to tell the system where to look for those files?

Or any other ideas why it always an empty string while there is clearly text on the image?

Thx for the help!

[dmariogatto]

Hi all,

I've create updated bindings for the following SDKs, at the time of posting this includes,

• Text Recognition v2
• Face Detection
• Barcode Scanning
• Image Labeling
• Object Detection & Tracking
• Digital Ink Recognition

MLKit Vision Bindings MLKit Vision Sample

I'll probably merge these into branch for #501.

@deelw Could you please reopen this issue? Thanks!

Also tagging #431, as it's a related issue.

[BjornVanslembrouck]

@dmariogatto, thanks for this!!

I initially had some issues building the --targets=externals, because Firebase.CloudFirestore had some issues with generating the SSL package bindings in one of its deeper dependencies. As I don't really need those, I commented them out in the cake files and I could then without issues generate the nuget packages.

However, when using them, I can reference all the methods from my project, but when running it on a simulator I keep getting the following error: Could not create an native instance of the type 'MLKit.TextRecognition.LatinTextRecognizerOptions': the native class hasn't been loaded. It is possible to ignore this condition by setting ObjCRuntime.Class.ThrowOnInitFailure to false

After some extra digging and comparing with your samples application (which does work), I think the issue lays around the xamarin.build.download integrations in the target files. I think it should download these files, but I have the impression it just ignores them which then leads to the error that the native libraries aren't there.

Do you have any idea why these could be ignored when calling it from within a NuGet package? I tried loosening up the conditions on the <ItemGroup Condition="('$(OutputType)'!='Library' OR '$(IsAppExtension)'=='True')"> that surrounds the <XamarinBuildDownload Include="$(_GoogleAppMeasurementItemsFolder)"> tag as I thought it had something to do with my project type, but that didn't seem to work either.

[dmariogatto]

@BjornVanslembrouck

I haven't had a build error for the SSL package, was it XCode? I assume it was BoringSSL-GRPC?

The missing native class definitely sounds like the targets file isn't running. A trick to see if the target is actually executing is to add a Message element inside the Target (example), it'll display in the build output. I'd double check your iOS project imports and whether your walking up the correct number of directories.

[BjornVanslembrouck]

@dmariogatto, that is what confused me quite a bit TBH. I always assumed that when a .target file lives inside the build folder inside the NuGet package, it gets added to the project automatically. Read a bit more about it and it indeed seems that that is what it should do. There are however some articles mentioning that you need to add the targets and props files specifically to the nuspec, which isn't the case at the moment. Not sure if adding it to the nuspec file is for the old config file approach or for the packageReference as well.

I added the following lines to my .csproj and it now seems to work as expected: `

<Import Project="$(NuGetPackageRoot)xamarin.mlkit.ios.textrecognition/1.0.0/buildTransitive/Xamarin.MLKit.iOS.TextRecognition.targets" />
<Import Project="$(NuGetPackageRoot)Xamarin.MLKit.iOS.TextRecognition.Latin/1.4.0/buildTransitive/Xamarin.MLKit.iOS.TextRecognitionLatin.targets" />
<Import Project="$(NuGetPackageRoot)Xamarin.MLKit.iOS.VisionKit/3.0.0/buildTransitive/Xamarin.MLKit.iOS.VisionKit.targets" />`

Thanks for the help!!

[deelw]

@dmariogatto for iOS we did switch to Apple Vision for OCR. Would you still want me to reopen this issue?

[dmariogatto]

@deelw Thanks for following up. I wouldn't worry about reopening the issue.

[deelw]

@dmariogatto I would not reopen this issues unless you request me to. And you did so Nov 21th 2021 by tagging me "@deelw Could you please reopen this issue? Thanks!" Do you still want me to reopen this issue?

[dmariogatto]

@deelw Apologies, I forgot about my previous request. Leave it closed for the moment, it's up to the project maintainers if they want it reopened.