[firebase_ml_vision] Text is either not detected or scrambled on iOS (Android is working fine!)

[TehRaab]

Describe the bug When taking photo of some text, no text is recognised. This seems to only be an issue on iOS devices. For example: On an iPhone 6 the text is not recognised at all in portrait, but OCR works sometimes in landscape. There is also an issue with iPad mini 2 where text is either not recognised or it is mirrored/flipped! I think the image capture from iOS changes the orientation of the saved image or exif data is messed up! I have not seen any of these issues when testing on Android devices.

I have the podfile setup to include 'Firebase/MLVisionTextModel'. This works fine on Android and only appears to be an issue on iOS which is now preventing me from releasing my app.

I had also attempted to use the flutter_image_compress but this seemed to make things worse and also break the Android functionality and as such the flutter_image_compress checks i tried are not included in the source below.

To Reproduce I've put together this mini test app. I had a postit and simply wrote "Hello" without quotes on it. Then run this app on and iPhone 6 and iPad Mini 2 in both landscape and portrait and took a photo of it. Once done the app will display the photo that was taken along with the text that was detected. There is also some debug comments in the code to spit out what is seen

pubspec.yaml added plugins firebase_ml_vision: ^0.9.4 image_picker: ^0.6.7+1 path: path_provider: ^1.6.7

main.dart ` import 'dart:io'; import 'package:firebase_ml_vision/firebase_ml_vision.dart'; import 'package:flutter/material.dart'; import 'package:image_picker/image_picker.dart'; import 'package:path_provider/path_provider.dart' as syspaths; import 'package:path/path.dart' as path;

void main() => runApp(MyApp());

class MyApp extends StatefulWidget {
    @override
    _MyAppState createState() => _MyAppState();
}

class _MyAppState extends State<MyApp> {
    File _image;
    String _detectedText = '';

    Future<String> _takePicture() async {
        setState(() {
            _image = null;
        });

        final _picker = ImagePicker();
        final PickedFile image = await _picker.getImage(
            source: ImageSource.camera,
            imageQuality: 100,
        );

        final File imageFile = File(image.path);

        if (imageFile == null) {
            return null;
        }

        final appDir = await syspaths.getApplicationDocumentsDirectory();
        final fileName = path.basename(imageFile.path);

        final savedImage = await imageFile.copy('${appDir.path}/$fileName');
        return '${appDir.path}/$fileName';
    }

    void _cameraScanSearch() async {
        print('_cameraScanSearch()');
        _image = null;

        final imagePath = await _takePicture();
        if (imagePath == null || imagePath.isEmpty) {
            print('_cameraScanSearch() - imagePath is null');
            return;
        }

        setState(() {
            _image = File(imagePath);
        });

        try {
            var stopWatch = Stopwatch()..start();

            print('_cameraScanSearch() - Camera Scan..');
            final FirebaseVisionImage visionImage =
            FirebaseVisionImage.fromFilePath(imagePath);

            final TextRecognizer _recognizer = FirebaseVision.instance.textRecognizer();
            final VisionText readText = await _recognizer.processImage(visionImage);
            setState(() {
                _detectedText = readText.text;
            });

            print('_cameraScanSearch() - VisionText found -> ${readText.text}');

            var blocks = readText.blocks;
            if (blocks.length == 0) {
                print('_cameraScanSearch() - blocks.length == 0');
            }

            for (TextBlock block in blocks) {
                print('_cameraScanSearch() - block -> $block');
            }
            print('_cameraScanSearch() - search took: ${stopWatch.elapsed}');
            stopWatch.stop();
            _recognizer.close();
        } catch (error) {
            print('_cameraScanSearch() - error: $error');
        }
    }

    @override
    Widget build(BuildContext context) {
        return MaterialApp(
            home: Scaffold(
                appBar: AppBar(
                    title: Text(
                        'Firebase_ML_Vision - MPV',
                        style: TextStyle(
                            fontSize: 24,
                            fontWeight: FontWeight.w400,
                        ),
                    ),
                ),
                body: SingleChildScrollView(
                    child: Container(
                        child: Column(
                            mainAxisSize: MainAxisSize.max,
                            children: <Widget>[
                                Center(
                                    child: FlatButton.icon(
                                        onPressed: _cameraScanSearch,
                                        icon: Icon(Icons.camera_alt),
                                        label: Text('Take Photo'),
                                    ),
                                ),
                            Text (
                                'Detected Text: $_detectedText'
                            ),

                            if(_image != null)
                                Image.file(_image),
                        ],
                    ),
                ),
            ),
        ), //Scaffold
    );
}}`

Expected behavior Firebase ML Vision Text should be detecting text in all orientations

Actual behaviour Text is recognised correctly on Android in both portrait and landscape, however for iOS text is not detected. There may actually be two issues here. 1) Text is not recognised in Portrait, but is detected when the picture is taken in landscape. 2) On iPad Mini 2, text is not recognised and when outputting text it seems to be mirrored and garbage! I think the photo orientation is off when taking photos in iOS devices.

Taking a photo of the word "Hello" on an iPhone 6.. Portrait detects no text. Landscape detects "jjsh"

Additional context Add any other context about the problem here.

Flutter doctor Note that although it complains about the Flutter and Dart plugins, it is launching and running (i think this is a separate issue with flutter doctor and Intellij).

flutter doctor -v [✓] Flutter (Channel stable, v1.17.3, on Mac OS X 10.15.4 19E287, locale en-GB) • Flutter version 1.17.3 at /Users/rob/development/flutter • Framework revision b041144f83 (12 days ago), 2020-06-04 09:26:11 -0700 • Engine revision ee76268252 • Dart version 2.8.4

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2) • Android SDK at /Users/rob/Library/Android/sdk • Platform android-29, build-tools 29.0.2 • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593) • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.5) • Xcode at /Applications/Xcode.app/Contents/Developer • Xcode 11.5, Build version 11E608c • CocoaPods version 1.9.1

[✓] Android Studio (version 4.0) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin version 46.0.2 • Dart plugin version 193.7361 • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)

[!] IntelliJ IDEA Community Edition (version 2020.1.1) • IntelliJ at /Applications/IntelliJ IDEA CE.app ✗ Flutter plugin not installed; this adds Flutter specific functionality. ✗ Dart plugin not installed; this adds Dart specific functionality. • For information about installing plugins, see https://flutter.dev/intellij-setup/#installing-the-plugins

[✓] Connected device (1 available) • iPhone 6 • f0c4a8bb34aada27a546e68d7646014db2ca8dc9 • ios • iOS 12.3.1

! Doctor found issues in 1 category.

[TahaTesser]

Issue reproducible on iOS and not on Android

flutter doctor -v

``` [✓] Flutter (Channel dev, 1.20.0-0.0.pre, on Mac OS X 10.15.5 19F101, locale en-GB) • Flutter version 1.20.0-0.0.pre at /Users/tahatesser/Code/flutter_dev • Framework revision d9653445f4 (6 days ago), 2020-06-09 18:43:03 -0400 • Engine revision e8c13aa012 • Dart version 2.9.0 (build 2.9.0-14.0.dev 5c1376615e) [✓] Android toolchain - develop for Android devices (Android SDK version 30.0.0) • Android SDK at /Users/tahatesser/Code/sdk • Platform android-30, build-tools 30.0.0 • ANDROID_HOME = /Users/tahatesser/Code/sdk • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS (Xcode 11.5) • Xcode at /Applications/Xcode.app/Contents/Developer • Xcode 11.5, Build version 11E608c • CocoaPods version 1.9.3 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] Android Studio (version 4.0) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin version 46.0.2 • Dart plugin version 193.7361 • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593) [✓] VS Code (version 1.46.0) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.11.0 [✓] Connected device (4 available) • SM M305F • 32003c30dc19668f • android-arm64 • Android 10 (API 29) • macOS • macOS • darwin-x64 • Mac OS X 10.15.5 19F101 • Web Server • web-server • web-javascript • Flutter Tools • Chrome • chrome • web-javascript • Google Chrome 83.0.4103.97 • No issues found! ```

[sharecube]

The same issue. Also iPhone 6, iOS 12.4. On Android works fine.

[TehRaab]

May i ask would the road-map/time frame usually is for a bug fix like this? I have had to postpone the release of my iOS app due to this issue and only release to google play (android is working great). I haven't been able to come up with a viable temporary solution for this iOS issue - which is very frustrating.

[magnuswikhog]

@TehRaab Did you try using flutter_image_compress to recompress the image on iOS? It seems to have solved the problems I was having on iOS, where text would only be detected in landscape orientation.

pubspec.yaml flutter_image_compress: ^0.7.0

code

// Recompress _image to fix orientation/text recognition on iOS
Uint8List compressedBytes = await FlutterImageCompress.compressWithFile(_image.path, keepExif: true);
_image = await _image.writeAsBytes(compressedBytes, flush: true);

[TehRaab]

@TehRaab Did you try using flutter_image_compress to recompress the image on iOS? It seems to have solved the problems I was having on iOS, where text would only be detected in landscape orientation.

pubspec.yaml flutter_image_compress: ^0.7.0

code

// Recompress _image to fix orientation/text recognition on iOS
Uint8List compressedBytes = await FlutterImageCompress.compressWithFile(_image.path, keepExif: true);
_image = await _image.writeAsBytes(compressedBytes, flush: true);

Thank you. I did try this package a while back but must have had something wrong as it works with your method :) I do wonder why the compression help!?

[TehRaab]

@TehRaab Did you try using flutter_image_compress to recompress the image on iOS? It seems to have solved the problems I was having on iOS, where text would only be detected in landscape orientation. pubspec.yaml flutter_image_compress: ^0.7.0 code

// Recompress _image to fix orientation/text recognition on iOS
Uint8List compressedBytes = await FlutterImageCompress.compressWithFile(_image.path, keepExif: true);
_image = await _image.writeAsBytes(compressedBytes, flush: true);

Thank you. I did try this package a while back but must have had something wrong as it works with your method :) I do wonder why the compression help!?

Update - i have had reports from a friend that even with this work around, the issue is still seen. Landscape is simply not working on his iPhone (X i believe)

[eliaweiss]

I was not able to solve the problem with flutter_image_compress

[Salakar]

Hey, :wave:, the firebase_ml_vision package is now discontinued since its APIs have been deprecated and removed from the Android & iOS Firebase SDKs.

I'd recommend switching to the alternatives now;

• for on-device vision APIs; switch to Google's standalone ML Kit library via google_ml_kit
• for cloud vision APIs; the recommended approach now is to use Firebase Authentication and Functions, which gives you a managed, serverless gateway to the Google Cloud Vision APIs. For an example Functions project see the vision-annotate-images sample project. You'd deploy this project for example and then call the cloud function from your app with your image data to get the annotation result.

Apologies for any inconvenience here and best of luck switching over the the new APIs. firebase_ml_custom is not affected by this deprecation.

Thanks