[firebase_ml_vision] textRecognizer doesn't work on iOS (no cloudTextRecognizer workaround)

[Alecsplus]

Describe the bug Same bug of issue 1276 https://github.com/FirebaseExtended/flutterfire/issues/1276

Also in my case I have this problem only on iOS devices. Text is recognized only with cloudTextRecognizer and not with textRecognizer method (with the same image).

This is a really great problem, because cloud recognition needs billing.

What we can do to to solve it?

Thanks in advance

To Reproduce use textRecognizer() on iOS devices do not works. cloudTextRecognizer() works well

Expected behavior textRecognmizer should give a similar result to cloudTextRecognizer

Additional context only iOS. Android works well.

[iapicca]

Hi @Alecsplus can you please provide your flutter doctor -v , your flutter run --verbose and your pubspec.yaml or if possible a reproducible minimal code sample. Thank you

[chaser79]

The FirebaseVision.instance.textRecognizer() is not recognizing any text on iOS device. It works perfect on an android device.

To Reproduce Use FirebaseVision.instance.textRecognizer() on an image to try and recognize text.

void _read() {
    progressDialog.show();
    _takePicture().then((String filePath) async {
        final FirebaseVisionImage visionImage =
            FirebaseVisionImage.fromFilePath(filePath);
        final TextRecognizer textRecognizer =
            FirebaseVision.instance.textRecognizer();
        textRecognizer.processImage(visionImage).then((visionText) {
          print(visionText.text);
          _processText(visionText.blocks);
          textRecognizer.close();
        }).catchError((error) {
          progressDialog.dismiss();
          PlatformErrorAlert(context: context, text: error).show();
        });
    }).catchError((error) {
      progressDialog.dismiss();
      PlatformErrorAlert(context: context, text: error).show();
    });
  }

The print(visionText.text); is empty when using an iOS device.

I expect it to return a VisionText with the recognized text like it does on an android device.

Flutter doctor output

Doctor summary:
[✓] Flutter (Channel master, v1.16.4-pre.22, on Mac OS X 10.14.6 18G103, locale en-GB)
    • Flutter version 1.16.4-pre.22 at /Users/lukechase/Apps/FlutterSDK
    • Framework revision e4a61dfdea (12 hours ago), 2020-03-29 21:01:02 -0400
    • Engine revision 361d5d5320
    • Dart version 2.8.0 (build 2.8.0-dev.17.0 1402e8e1a4)

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
    • Android SDK at /Users/lukechase/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-29, build-tools 29.0.3
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_212-release-1586-b4-5784211)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.3.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.3.1, Build version 11C504
    • CocoaPods version 1.9.1

[!] Android Studio (version 3.6)
    • Android Studio at /Applications/Android Studio.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_212-release-1586-b4-5784211)

[✓] VS Code (version 1.43.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.8.1

[✓] Connected device (1 available)
    • iPhone XR • 00008020-001D6D883C45002E • ios • iOS 13.3.1

! Doctor found issues in 1 category.

[zelliot]

On Android textRecognizer() and cloudTextRecognizer() both work perfectly.

On iOS neither textRecognizer() or cloudTextRecognizer() work.

I've done a bit of digging on this and I think it is something to do with the image. It seems to be rotated or flipped on iOS.

Testing with a photo of my Australian Passport taken within the app on iOS, textRecognizer doesn't work at all but with cloudTextRecognizer it seems to work but returns everything back to front i.e. "AIJAЯТ2UA"

Flutter doctor output: [✓] Flutter (Channel stable, v1.12.13+hotfix.9, on Mac OS X 10.15.4 19E266, locale en-AU) • Flutter version 1.12.13+hotfix.9 at /Users/elliot/dev/flutter • Framework revision f139b11009 (9 days ago), 2020-03-30 13:57:30 -0700 • Engine revision af51afceb8 • Dart version 2.7.2

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2) • Android SDK at /Users/elliot/Library/Android/sdk • Android NDK location not configured (optional; useful for native profiling support) • Platform android-29, build-tools 29.0.2 • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405) • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.4) • Xcode at /Applications/Xcode.app/Contents/Developer • Xcode 11.4, Build version 11E146 • CocoaPods version 1.9.1

[✓] Android Studio (version 3.5) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin version 43.0.1 • Dart plugin version 191.8593 • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)

[✓] VS Code (version 1.43.2) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.9.0

[✓] Connected device (1 available) • ePhone • 47e705ac2711c7e3436bbe0fda4b9ab3beeafbef • ios • iOS 13.4

• No issues found!

[lurajon]

Hi! I experience the same problem;

`void _initializeVision() async { final File imageFile = File(path);

if (imageFile != null) {
  await _getImageSize(imageFile);
}

await GallerySaver.saveImage(path);

final FirebaseVisionImage visionImage =
    FirebaseVisionImage.fromFile(imageFile);

final TextRecognizer textRecognizer =
    FirebaseVision.instance.textRecognizer();

print(visionImage);
final VisionText visionText =
    await textRecognizer.processImage(visionImage);

String text = "";
print(visionText.text);
for (TextBlock block in visionText.blocks) {
  print(block.text);
  for (TextLine line in block.lines) {
      print(line.text);
      text += line.text + '\n';
      for (TextElement element in line.elements) {
        print(element.text);
        _elements.add(element);
      }

  }
}

if (this.mounted) {
  setState(() {
    recognizedText = text;
  });
}

}`

pubspec.yml

When I open a camerastream however, and use fromBytes

FirebaseVisionImage.fromBytes( _concatenatePlanes(image.planes), _buildMetaData(image, _rotationIntToImageRotation(imageRotation)), ), Text are recognized on the stream.

Maybe the issue is somewhere in FirebaseVisionImage.fromFile(imageFile);?

[junnyea]

Facing the same problem here. Value for visionText is empty

VisionText visionText = await textRecognizer.processImage(visionImage)

[hariprasadiit]

For me with original photo it doesn't work but, If I use flutter_image_compress and use the resulted compressed image, on device text recognition works well. I think it has something to do with rotation and EXIF information.

[TehRaab]

For me with original photo it doesn't work but, If I use flutter_image_compress and use the resulted compressed image, on device text recognition works well. I think it has something to do with rotation and EXIF information.

I am also facing the same problem. If i am to use flutter_image_compress, how do you determine how much rotation is needed? I have noticed that the iPad Mini 2 only works in landscape. I believe this is because the camera hardware is actually physically installed at +/-90 degree angle.

[hariprasadiit]

@TehRaab I think flutter_image_compress automatically rotates the images based on EXIF information

[nickykln]

Hello, any progress on that issue ? @hariprasadiit, I have try the flutter_image_compress workaround with no success. Could you please share the piece of code you wrote as a workaround.

Thanks a lot for your help,

Nicolas

[robwafle-scripd]

Hi! We are experiencing this issue too. And, we're starting to have some issues with Android as well. Usually, might get an exception or two when starting the android app saying something along the lines of "please wait, still downloading ocr model from cloud", but now it won't download at all.

Also, I am getting some warnings when building my app: [!] FirebaseMLVisionTextModel has been deprecated in favor of MLKitTextRecognition

It looks like the MLKit is going through a bit of a transformation at the moment: https://firebase.google.com/docs/ml-kit/recognize-text

This appears to be the pod file name that will replace the current podfile: pod 'GoogleMLKit/TextRecognition'

But, replacing with this line didn't help either.

[SzymonDziak]

Android works perfectly for me. iOS text detection does not work. It detects weird texts and outputs wrong information if it pick up something.

[TehRaab]

Android works perfectly for me. iOS text detection does not work. It detects weird texts and outputs wrong information if it pick up something.

I've had this issue for several months now and still haven't found a solution. I've you find one, please let us know. I raised a separate issue back in June - https://github.com/FirebaseExtended/flutterfire/issues/2770

[a-arshad]

Has there been any updates or other workarounds on this problem?

The flutter_image_compress used to be fine as a workaround for a while, but now it seems like nothing is being detected on iOS at all. Has the cloudTextRecognizer been working recently as well, or is that still broken?

[SzymonDziak]

Hello everyone. If you are having problems with Image Text Recognition for iOS I found the fix. After testing around multiple times, and you can try this to the text detection works but it just detects text incorrectly. Why is that? Well if you start fliping the phone around and image you will realize the issue becomes with the orientation of the screen. The fix then lays in fixing the orientation of the screen for iOS.

ML Vision uses pictures (hundreds of them per second) to detect whatever you want. You have to orient each image to the correct space and then send it to the text detector.

This is the plugin I used: flutter_exif_rotation:

[SzymonDziak]

So you await to take a picture. You send the path of the image into the Flutter Exif Rotation and then send it to Firebase Vision. Hope this helps.

It took me a few days of pulling my hair out to figure this out.

[a-arshad]

Hello everyone. If you are having problems with Image Text Recognition for iOS I found the fix. After testing around multiple times, and you can try this to the text detection works but it just detects text incorrectly. Why is that? Well if you start fliping the phone around and image you will realize the issue becomes with the orientation of the screen. The fix then lays in fixing the orientation of the screen for iOS.

ML Vision uses pictures (hundreds of them per second) to detect whatever you want. You have to orient each image to the correct space and then send it to the text detector.

This is the plugin I used: flutter_exif_rotation:

I used to have this problem and so if I were to orient the device differently, the text recognizer would return text correctly. However, even with your fix or using flutter_compress_image, this is not the case. The FlutterVisionText returns no text read for any orientation.

if (Platform.isIOS) {
      file = await FlutterExifRotation.rotateImage(path: path);
}

FirebaseVisionImage image = FirebaseVisionImage.fromFile(file);

[a-arshad]

Just an update, this workaround still works. The error I was having was because I was using the new GoogleMLKit/TextRecognition pod. Instead if you just use the deprecated pods, Firebase/MLVision and Firebase/MLVisionTextModel, the textRecognizer works fine. Hopefully the newer pod gets patched up to work with Flutter soon.

[rohan20]

I can confirm that the solution proposed by @SimonDziak and its example from @a-arshad works with on-device text recognizer (didn't try the cloud one).

if (Platform.isIOS) {
      file = await FlutterExifRotation.rotateImage(path: path);
}

FirebaseVisionImage image = FirebaseVisionImage.fromFile(file);

// pubspec.yaml
flutter_exif_rotation: ^0.3.2

Thank you both of you! ✅

[bmvkrd]

I can also confirm that the above solution proposed by @SimonDziak works (when FlutterExifRotation is used). Thanks a lot, guys!

[eliaweiss]

@SimonDziak fix did not work for me ;(

[martinsmoises]

it didn't work for me either.

[martinsmoises]

Add 'Firebase/MLVisionTextModel' on Podfile was work for me

pod Firebase/MLVisionTextModel

[Salakar]

Hey, :wave:, the firebase_ml_vision package is now discontinued since its APIs have been deprecated and removed from the Android & iOS Firebase SDKs.

I'd recommend switching to the alternatives now;

• for on-device vision APIs; switch to Google's standalone ML Kit library via google_ml_kit
• for cloud vision APIs; the recommended approach now is to use Firebase Authentication and Functions, which gives you a managed, serverless gateway to the Google Cloud Vision APIs. For an example Functions project see the vision-annotate-images sample project. You'd deploy this project for example and then call the cloud function from your app with your image data to get the annotation result.

Apologies for any inconvenience here and best of luck switching over the the new APIs. firebase_ml_custom is not affected by this deprecation.

Thanks

[zelliot]

@Salakar, is google_ml_kit an official package?

Also, the google_ml_kit only seems to support Android.