Xamarin.GooglePlayServices.MLKit.Text.Recognition doesn't contain expected APIs

[ivanlabsii]

Xamarin.Android Version (eg: 6.0):

12.3.3.3

Operating System & Version (eg: Mac OSX 10.11):

Windows 10, 19044.1826

Google Play Services Version

• [] GPS-FB with AndroidX dependencies (versions 1xx.yyyy.zz) Xamarin.AndroidX.AppCompat 1.4.2.1

Describe your Issue

I am trying to implement this: https://developers.google.com/ml-kit/vision/text-recognition/android

They suggest use of the com.google.android.gms:play-services-mlkit-text-recognition:18.0.0. After some help in https://github.com/xamarin/GooglePlayServicesComponents/issues/660#issuecomment-1189565158 I have found the package with that title here: https://www.nuget.org/packages/Xamarin.GooglePlayServices.MLKit.Text.Recognition

I have tried to add this line with that package: var recognizer = Xamarin.Google.MLKit.Vision.Text.TextRecognition.GetClient();

It is showing that Xamarin.Google namespace doesn't exist. Also there is no intellisense namespace suggestion for MLKitor for TextRecognition.

I was suggested in https://github.com/xamarin/GooglePlayServicesComponents/issues/660#issuecomment-1189565158 to use this package: https://www.nuget.org/packages/Xamarin.GooglePlayServices.MLKit.Text.Recognition.Common . Personally I'd be happy to use that package as an alternative but it doesn't work either. But even if it worked this is an issue as this package is clearly not what Google suggests to be used and at the end the package that Google suggests should be made to work too (and there is no reason to have it on the nuget if it doesn't do what is intended).

Relevant information

Add relevant project settings from *.csproj file:

Packages used:

    <PackageReference Include="Xamarin.Essentials" Version="1.7.3" />

Build settings (tools)

    <AndroidDexTool>d8</AndroidDexTool>
    <AndroidLinkTool>r8</AndroidLinkTool>
    <AndroidUseAapt2>true</AndroidUseAapt2>
    <AndroidEnableDesugar>true</AndroidEnableDesugar>

Minimal Repro Code Sample

App1.zip

If you want to speed up investigation and bug fixing: please provide minimal repro sample for tests.

Steps to Reproduce (with link to sample solution if possible):

Include any relevant Exception Stack traces, build logs, adb logs:

[ivanlabsii]

@jpobst just to note that I have searched through the class view and definitely nothing with MlKit is being exposed both with the correct package and the package that you have suggested.

It is almost funny that this is published then downloaded more than 60k times and no one (!) reported the issue though it seems to exists in every version.

[moljac]

Thanks for the feedback. Appreciated.

This link:

https://developers.google.com/ml-kit/vision/text-recognition/android

seems to be MLKit not GPS-mlkit. Google has different implementations for ML, some cloud based and some on-device (MLKit). We are no SDK and/or SDK API experts (though would like to have some time to play with those APIs).

Your link leads you to:

https://developers.google.com/ml-kit/vision/text-recognition/android

and your dependencies which you need to map to nuget packages are here:

https://github.com/googlesamples/mlkit/blob/master/android/vision-quickstart/app/build.gradle#L54-L140

dependencies {
    implementation 'org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.5.31'
    implementation 'androidx.multidex:multidex:2.0.1'

    // Barcode model
    implementation 'com.google.mlkit:barcode-scanning:17.0.2'
    // Or comment the dependency above and uncomment the dependency below to
    // use unbundled model that depends on Google Play Services
    // implementation 'com.google.android.gms:play-services-mlkit-barcode-scanning:18.0.0'

    // Object detection feature with bundled default classifier
    implementation 'com.google.mlkit:object-detection:17.0.0'

    // Object detection feature with custom classifier support
    implementation 'com.google.mlkit:object-detection-custom:17.0.0'

    // Face features
    implementation 'com.google.mlkit:face-detection:16.1.5'
    // Or comment the dependency above and uncomment the dependency below to
    // use unbundled model that depends on Google Play Services
    // implementation 'com.google.android.gms:play-services-mlkit-face-detection:17.0.1'

    // Text features
    implementation 'com.google.mlkit:text-recognition:16.0.0-beta4'
    // Or comment the dependency above and uncomment the dependency below to
    // use unbundled model that depends on Google Play Services
    // implementation 'com.google.android.gms:play-services-mlkit-text-recognition:18.0.0'
    implementation 'com.google.mlkit:text-recognition-chinese:16.0.0-beta4'
    implementation 'com.google.mlkit:text-recognition-devanagari:16.0.0-beta4'
    implementation 'com.google.mlkit:text-recognition-japanese:16.0.0-beta4'
    implementation 'com.google.mlkit:text-recognition-korean:16.0.0-beta4'

    // Image labeling
    implementation 'com.google.mlkit:image-labeling:17.0.7'
    // Or comment the dependency above and uncomment the dependency below to
    // use unbundled model that depends on Google Play Services
    // implementation 'com.google.android.gms:play-services-mlkit-image-labeling:16.0.7'

    // Image labeling custom
    implementation 'com.google.mlkit:image-labeling-custom:17.0.1'
    // Or comment the dependency above and uncomment the dependency below to
    // use unbundled model that depends on Google Play Services
    // implementation 'com.google.android.gms:play-services-mlkit-image-labeling-custom:16.0.0-beta3'

    // Pose detection with default models
    implementation 'com.google.mlkit:pose-detection:18.0.0-beta2'
    // Pose detection with accurate models
    implementation 'com.google.mlkit:pose-detection-accurate:18.0.0-beta2'

    // Selfie segmentation
    implementation 'com.google.mlkit:segmentation-selfie:16.0.0-beta4'

    implementation 'com.google.mlkit:camera:16.0.0-beta3'

    // -------------------------------------------------------

    implementation 'com.google.code.gson:gson:2.8.6'
    implementation 'com.google.guava:guava:27.1-android'

    // For how to setup gradle dependencies in Android X, see:
    // https://developer.android.com/training/testing/set-up-project#gradle-dependencies
    // Core library
    androidTestImplementation 'androidx.test:core:1.4.0'

    // AndroidJUnitRunner and JUnit Rules
    androidTestImplementation 'androidx.test:runner:1.4.0'
    androidTestImplementation 'androidx.test:rules:1.4.0'

    // Assertions
    androidTestImplementation 'androidx.test.ext:junit:1.1.3'

    // ViewModel and LiveData
    implementation "androidx.lifecycle:lifecycle-livedata:2.3.1"
    implementation "androidx.lifecycle:lifecycle-viewmodel:2.3.1"

    implementation 'androidx.appcompat:appcompat:1.2.0'
    implementation 'androidx.annotation:annotation:1.2.0'
    implementation 'androidx.constraintlayout:constraintlayout:2.0.4'

    // CameraX
    implementation "androidx.camera:camera-camera2:1.0.0-SNAPSHOT"
    implementation "androidx.camera:camera-lifecycle:1.0.0-SNAPSHOT"
    implementation "androidx.camera:camera-view:1.0.0-SNAPSHOT"

    // On Device Machine Learnings
    implementation "com.google.android.odml:image:1.0.0-beta1"
}

I will check the sample. I hope soon.

Thanks

[ivanlabsii]

We are no SDK and/or SDK API experts (though would like to have some time to play with those APIs).

I do understand this limitation. I do understand the huge scope of this work and that you generally can't test things.

This link:

https://developers.google.com/ml-kit/vision/text-recognition/android

seems to be MLKit not GPS-mlkit.

Reading just the header at the very top of the page it says:

Library name | com.google.android.gms:play-services-mlkit-text-recognition

Very little below that there is this code:

dependencies {
  // ...

  implementation 'com.google.android.gms:play-services-mlkit-text-recognition:18.0.0'
}

I am very open to the possibility that this is some other package than the one that I have referred. I was actually already opened to that possibility in https://github.com/xamarin/GooglePlayServicesComponents/issues/660#issuecomment-1189565158 which referred to this and the text seems to confirm that.

[ivanlabsii]

Looking further into the gradle this part seems to be key:

    // Text features
    implementation 'com.google.mlkit:text-recognition:16.0.0-beta4'
    // Or comment the dependency above and uncomment the dependency below to
    // use unbundled model that depends on Google Play Services
    // implementation 'com.google.android.gms:play-services-mlkit-text-recognition:18.0.0'

Basically for some reasons uses beta version (which likely was not bound yet) and suggest use of GPS ml-kit as an alternative, so it seems that they are interchangeable. However in the documentation and codelabs they refer only to GPS version, so I guess that in some future they might move it to the new place again but currently GPS is the one that we should be looking for.

[ivanlabsii]

But to conclude the current GPS MLKit package doesn't seem to contain anything MLKit related (Class view shows that nothing is added in the MLKit space) and I believe that it should resolve my problem if it worked well. I asked here for it to be fixed as it definitely needs to be fixed. If it turns out that that doesn't resolve my problem I guess I'll have to spend more time on it :). But let's fix this if you see that this is the bug in this package and I am 100% certain it is.

[moljac]

TextRecognition does exist in GPS-MlKit bindings Xamarin.Google.MLKit.Vision.Text.TextRecognition

ls -1 ./generated/com.google.android.gms.play-services-mlkit-text-recognition-common/obj/Release/net6.0-android31.0/generated/src/

result:

Java.Interop.__TypeRegistrations.cs
Xamarin.Google.MLKit.Vision.Text.ITextRecognizer.cs
Xamarin.Google.MLKit.Vision.Text.ITextRecognizerOptionsInterface.cs
Xamarin.Google.MLKit.Vision.Text.Internal.LoggingUtils.cs
Xamarin.Google.MLKit.Vision.Text.Internal.TextRecognizerImpl.cs
Xamarin.Google.MLKit.Vision.Text.Internal.TextRecognizerTaskWithResource.cs
Xamarin.Google.MLKit.Vision.Text.Internal.TextRegistrar.cs
Xamarin.Google.MLKit.Vision.Text.Text.cs
Xamarin.Google.MLKit.Vision.Text.TextRecognition.cs
Xamarin.GooglePlayServices.MLKit.Text.Recognition.Common.projitems
__NamespaceMapping__.cs
enumlist
generator.rsp

using System;
using System.Collections.Generic;
using Android.Runtime;
using Java.Interop;

namespace Xamarin.Google.MLKit.Vision.Text {

    // Metadata.xml XPath class reference: path="/api/package[@name='com.google.mlkit.vision.text']/class[@name='TextRecognition']"
    [global::Android.Runtime.Register ("com/google/mlkit/vision/text/TextRecognition", DoNotGenerateAcw=true)]
    public partial class TextRecognition : global::Java.Lang.Object {
        static readonly JniPeerMembers _members = new XAPeerMembers ("com/google/mlkit/vision/text/TextRecognition", typeof (TextRecognition));

        internal static IntPtr class_ref {
            get { return _members.JniPeerType.PeerReference.Handle; }
        }

        [global::System.Diagnostics.DebuggerBrowsable (global::System.Diagnostics.DebuggerBrowsableState.Never)]
        [global::System.ComponentModel.EditorBrowsable (global::System.ComponentModel.EditorBrowsableState.Never)]
        public override global::Java.Interop.JniPeerMembers JniPeerMembers {
            get { return _members; }
        }

        [global::System.Diagnostics.DebuggerBrowsable (global::System.Diagnostics.DebuggerBrowsableState.Never)]
        [global::System.ComponentModel.EditorBrowsable (global::System.ComponentModel.EditorBrowsableState.Never)]
        protected override IntPtr ThresholdClass {
            get { return _members.JniPeerType.PeerReference.Handle; }
        }

        [global::System.Diagnostics.DebuggerBrowsable (global::System.Diagnostics.DebuggerBrowsableState.Never)]
        [global::System.ComponentModel.EditorBrowsable (global::System.ComponentModel.EditorBrowsableState.Never)]
        protected override global::System.Type ThresholdType {
            get { return _members.ManagedPeerType; }
        }

        protected TextRecognition (IntPtr javaReference, JniHandleOwnership transfer) : base (javaReference, transfer)
        {
        }

        // Metadata.xml XPath method reference: path="/api/package[@name='com.google.mlkit.vision.text']/class[@name='TextRecognition']/method[@name='getClient' and count(parameter)=1 and parameter[1][@type='com.google.mlkit.vision.text.TextRecognizerOptionsInterface']]"
        [Register ("getClient", "(Lcom/google/mlkit/vision/text/TextRecognizerOptionsInterface;)Lcom/google/mlkit/vision/text/TextRecognizer;", "")]
        public static unsafe global::Xamarin.Google.MLKit.Vision.Text.ITextRecognizer GetClient (global::Xamarin.Google.MLKit.Vision.Text.ITextRecognizerOptionsInterface options)
        {
            const string __id = "getClient.(Lcom/google/mlkit/vision/text/TextRecognizerOptionsInterface;)Lcom/google/mlkit/vision/text/TextRecognizer;";
            try {
                JniArgumentValue* __args = stackalloc JniArgumentValue [1];
                __args [0] = new JniArgumentValue ((options == null) ? IntPtr.Zero : ((global::Java.Lang.Object) options).Handle);
                var __rm = _members.StaticMethods.InvokeObjectMethod (__id, __args);
                return global::Java.Lang.Object.GetObject<global::Xamarin.Google.MLKit.Vision.Text.ITextRecognizer> (__rm.Handle, JniHandleOwnership.TransferLocalRef);
            } finally {
                global::System.GC.KeepAlive (options);
            }
        }

    }
}

[moljac]

your code (from the submitted app sample):

            //Here is the problem
            var recognizer = Xamarin.Google.MLKit.Vision.Text.TextRecognition.GetClient();

does not have parameter!

        public static unsafe global::Xamarin.Google.MLKit.Vision.Text.ITextRecognizer GetClient (global::Xamarin.Google.MLKit.Vision.Text.ITextRecognizerOptionsInterface options)

so it is not that API is missing.

And I get intellisense error about missing parameter:

Error CS7036 There is no argument given that corresponds to the required formal parameter 'options'

[ivanlabsii]

does not have parameter!

You are right, but it was intended to, without namespace it was difficult to write a parameter and it was not necessary to show that there is no namespace.

I think that you missed this part:

Windows 10, 19044.1826

As you said that it works I did try on a Mac and it really works. Not sure why it didn't work on Windows in VS 2022.

[moljac]

You are right, but it was intended to, without namespace it was difficult to write a parameter and it was not necessary to show that there is no namespace.

"no namespace" doesn't have to be problem of bindings. Yes, package surfacing (generating MCWs) could fail, but this is rare case.

As you said that it works I did try on a Mac and it really works. Not sure why it didn't work on Windows in VS 2022.

I tried on Mac (and Mac is waaay behind Windows with respect to features and stability) and intellisense worked.

Not sure why it didn't work on Windows in VS 2022.

Installation, setup, ....

[ivanlabsii]

OK, I guess that I'll submit the issue to VS.

It is good to know that I can work on this on Mac which at least initially unblocks the problem for me. Thanks for taking the time.

[ivanlabsii]

OK, I can see what is likely to cause this weird behavior.

When you add this package to VS it can't even build the solution (even without any code), the error says: .nuget\packages\xamarin.build.download\0.11.2\buildTransitive\Xamarin.Build.Download.targets(52,3): error XBD020: Invalid item ID image-1.0.0-beta1

I'll check tomorrow on the Mac again but it is likely that there will be this same problem. Just it has different effects on Intellisense (and editor) on the Mac and on Windows.

[ivanlabsii]

I've tried it on the Mac. Commented that line and just tried to build the solution. It doesn't work (same error) and that was the cause of the problems that I have first encountered.

So it turns out that there is a bug and it needs to be fixed...

[moljac]

"same error" is completely different issue. Reopening something that was not issue from the beginning is not very helpful and causes us to waste our scarce resources.

So it turns out that there is a bug and it needs to be fixed...

This is repo for GPS, FB and MLKit bindings and bug was in Xamarin.Build.Download.

[moljac]

try updating (explicitly adding reference) to projects:

<PackageReference Include="Xamarin.Build.Download" Version="0.11.3" />

And please report if this works.

Thanks

[ivanlabsii]

try updating (explicitly adding reference) to projects:

<PackageReference Include="Xamarin.Build.Download" Version="0.11.3" />

And please report if this works.

Thanks

Thanks. This seems to resolve the problem for me. From that point of view, feel free to close the issue (I won't close it myself as at this won't work for other people until this is fixed somehow, whatever it requires).

Also I do understand that it might not be your direct responsibility but still there is no doubt that this doesn’t work at the moment.

[moljac]

Simple question: Does it work or not?

[ivanlabsii]

It does compile properly and inntellisense works too. So I don’t have any of the problems mentioned here after adding the Build package.

[moljac]

closing this one

[datoml]

I tried to track it further down and it seems it is coming from the MLKit itself :(.

When I remove this part the memory footprint looks good.