[firebase_ml_vision] Cloud Text Recognition returns wrong coordinates on iOS

[kroikie]

When using the new cloud text recognizer introduced in https://github.com/flutter/plugins/pull/1847, the coordinates of the recognized text are not correct on iOS (works on Android). However, text detection works fine.

The problem does not appear when using offline OCR on both platforms. Moreover, the problem with cloud OCR on IOS does not happen on every image. See attached screenshots.

Steps to Reproduce

1. Use FirebaseVision.instance.cloudTextRecognizer()
2. Progress image File
3. Look at coordinates

Supporting Code:

Future<VisionResult> analyze(File imageFile,
    {VisionMode visionMode = VisionMode.online}) {
  FirebaseVisionImage image = FirebaseVisionImage.fromFile(imageFile);

  TextRecognizer textRecognizer;
  if (visionMode == VisionMode.offline) {
    textRecognizer = FirebaseVision.instance.textRecognizer();
  } else {
    textRecognizer = FirebaseVision.instance.cloudTextRecognizer();
  }
  var result = textRecognizer.processImage(image).then((visionText) {
    return VisionResult(
        ingredientsAnalysis: analyzer.analyze(visionText),
        visionText: visionText);
  });

  return Future.sync(() async {
    await textRecognizer.close();
    return result;
  });
}

Coordinates code:

entry['corner_points'] = [
  [boundingBox.topLeft.dx, boundingBox.topLeft.dy],
  [boundingBox.topRight.dx, boundingBox.topRight.dy],
  [boundingBox.bottomRight.dx, boundingBox.bottomRight.dy],
  [boundingBox.bottomLeft.dx, boundingBox.bottomLeft.dy],
];

Vision API JSON Representation

Online OCR:

iOS

[[{"text":"Referenzmenge","corner_points":[[288.0,36.0],[534.0,36.0],[534.0,8.0],[288.0,8.0]]},{"text":"füreinen","corner_points":[[538.0,40.0],[675.0,40.0],[675.0,4.0],[538.0,4.0]]},{"text":"durchsch","corner_points":[[684.0,41.0],[809.0,41.0],[809.0,5.0],[684.0,5.0]]}],[{"text":"Erwachsenen","corner_points":[[353.0,66.0],[555.0,66.0],[555.0,33.0],[353.0,33.0]]},{"text":"(8400","corner_points":[[568.0,68.0],[653.0,68.0],[653.0,34.0],[568.0,34.0]]},{"text":"kJ","corner_points":[[663.0,67.0],[689.0,67.0],[689.0,37.0],[663.0,37.0]]},{"text":"/","corner_points":[[700.0,60.0],[710.0,60.0],[710.0,40.0],[700.0,40.0]]},{"text":"20o0k","corner_points":[[713.0,69.0],[820.0,69.0],[820.0,35.0],[713.0,35.0]]}],[{"text":"Ingesta","corner_points":[[310.0,97.0],[433.0,97.0],[433.0,61.0],[310.0,61.0]]},{"text":"de","corner_points":[[445.0,98.0],[481.0,98.0],[481.0,67.0],[445.0,67.0]]},{"text":"referencia","corner_points":[[490.0,101.0],[648.0,101.0],[648.0,65.0],[490.0,65.0]]}
...

Android

[[{"text":"Referenzmenge","corner_points":[[428.0,51.0],[758.0,51.0],[758.0,13.0],[428.0,13.0]]},{"text":"für","corner_points":[[766.0,55.0],[835.0,55.0],[835.0,6.0],[766.0,6.0]]},{"text":"einen","corner_points":[[840.0,56.0],[959.0,56.0],[959.0,6.0],[840.0,6.0]]},{"text":"durchsch","corner_points":[[970.0,58.0],[1148.0,58.0],[1148.0,7.0],[970.0,7.0]]}],[{"text":"Erwachsenen","corner_points":[[499.0,93.0],[788.0,93.0],[788.0,46.0],[499.0,46.0]]},{"text":"(8400","corner_points":[[806.0,93.0],[927.0,93.0],[927.0,46.0],[806.0,46.0]]},{"text":"kJ/","corner_points":[[934.0,93.0],[1018.0,93.0],[1018.0,46.0],[934.0,46.0]]},{"text":"2000","corner_points":[[1021.0,93.0],[1131.0,93.0],[1131.0,46.0],[1021.0,46.0]]}],[{"text":"Ingesta","corner_points":[[450.0,145.0],[610.0,145.0],[610.0,96.0],[450.0,96.0]]},{"text":"de","corner_points":[[641.0,138.0],[681.0,138.0],[681.0,90.0],[641.0,90.0]]},{"text":"referencia","corner_points":[[702.0,135.0],[911.0,135.0],[911.0,89.0],[702.0,89.0]]},
...

Offline OCR

iOS

 [[{"text":"ReferenzmengetUinen","corner_points":[[407.0,12.0],[960.9691772460938,17.844175338745117],[960.4944458007812,62.84166717529297],[406.5252990722656,56.997493743896484]]},{"text":"durchsch","corner_points":[[965.0,17.0],[1152.989501953125,18.98322105407715],[1152.5042724609375,64.98065948486328],[964.5147705078125,62.997440338134766]]}],[{"text":"Erwachsenen","corner_points":[[491.0,49.0],[796.9801635742188,52.48468780517578],[796.524658203125,92.48209381103516],[490.54449462890625,88.99740600585938]]},{"text":"(8400","corner_points":[[799.0,53.0],[936.9910888671875,54.57152557373047],[936.5355834960938,94.56893157958984],[798.5444946289062,92.99740600585938]]},{"text":"kj/","corner_points":[[939.0,55.0],[1023.9945068359375,55.96796798706055],[1023.5503540039062,94.96543884277344],[938.5558471679688,93.99746704101562]]},{"text":"2000k","corner_points":[[1026.0,56.0],[1156.991455078125,57.4918098449707],[1156.54736328125,96.4892807006836],[1025.555908203125,94.99746704101562]]}
...

Android

[[{"text":"'Referenzmenge","corner_points":[[384.0,0.0],[759.0,2.0],[758.0,51.0],[383.0,49.0]]},{"text":"für","corner_points":[[770.0,3.0],[842.0,3.0],[841.0,52.0],[769.0,52.0]]},{"text":"einen","corner_points":[[845.0,4.0],[967.0,4.0],[966.0,53.0],[844.0,53.0]]},{"text":"dur","corner_points":[[970.0,5.0],[1050.0,5.0],[1049.0,54.0],[969.0,54.0]]}],[{"text":"\"Ingesta","corner_points":[[407.0,74.0],[626.0,74.0],[626.0,149.0],[407.0,149.0]]},{"text":"de","corner_points":[[616.0,74.0],[688.0,74.0],[688.0,149.0],[616.0,149.0]]},{"text":"referena","corner_points":[[691.0,85.0],[919.0,85.0],[919.0,125.0],[691.0,125.0]]},{"text":"de","corner_points":[[900.0,91.0],[983.0,91.0],[983.0,130.0],[900.0,130.0]]},{"text":"u","corner_points":[[981.0,91.0],[1022.0,91.0],[1022.0,130.0],[981.0,130.0]]}],[{"text":"medio","corner_points":[[1189.0,116.0],[1309.0,116.0],[1309.0,163.0],[1189.0,163.0]]}],[{"text":"PATATAS","corner_points":[[896.0,569.0],[1076.0,570.0],[1075.0,613.0],[895.0,612.0]]},{"text":"FRITAS","corner_points":
...

Attachements

Original Image:

Coordinates visualization cloud on iOS. Rectangles are not placed correctly on text.

Coordinates visualization cloud on Android. Rectangles are placed correctly on text

Logs

[✓] Flutter (Channel stable, v1.7.8+hotfix.3, on Mac OS X 10.14.5 18F132, locale de-DE)
    • Flutter version 1.7.8+hotfix.3 at /Users/xxx/lib/flutter
    • Framework revision b712a172f9 (13 days ago), 2019-07-09 13:14:38 -0700
    • Engine revision 54ad777fd2
    • Dart version 2.4.0

[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at /Users/xxx/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-28, build-tools 28.0.3
    • ANDROID_HOME = /Users/xxx/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1136-b06)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 10.2.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 10.2.1, Build version 10E1001
    • CocoaPods version 1.7.1

[✓] iOS tools - develop for iOS devices
    • ios-deploy 1.9.4

[✓] Android Studio (version 3.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 31.3.1
    • Dart plugin version 181.5656
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1136-b06)

[✓] VS Code (version 1.36.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.2.0

[✓] Connected device (3 available)
    • GT I9195      • 0f3c8d5e                             • android-arm • Android 6.0.1 (API 23)
    • iPhone SE     • 57A511E2-A8D5-4726-8CAA-B8B877B5C88A • ios         • com.apple.CoreSimulator.SimRuntime.iOS-12-2 (simulator)
    • iPhone XS Max • CB708036-EB6F-404C-9BFF-612B05A2C4FE • ios         • com.apple.CoreSimulator.SimRuntime.iOS-12-2 (simulator)

• No issues found!

[kroikie]

@nachtmaar

The issue at https://github.com/flutter/flutter/issues/36705 has been closed and moved here. Future collaboration on this issue will be done here.

[iapicca]

Hi @nachtmaar If you are still experiencing this issue with the latest version of Flutter and FlutterFire plugin can you please provide your updated flutter doctor -v and your and your flutter run --verbose/flutter build --verbose ? Also, to better address the issue, would be helpful if you could post a self contained app on github or the steps to reproduce it. Thank you

[nachtmaar]

Yes, I am still experiencing the issue.

Sample code is at: https://github.com/nachtmaar/flutter-meetup-munich-ocr-demo/pull/1 Please see the PR for more information and let me know if you need more!

$ flutter doctor -v
[✓] Flutter (Channel stable, v1.9.1+hotfix.5, on Mac OS X 10.15 19A602, locale en-DE)
    • Flutter version 1.9.1+hotfix.5 at /Users/xxxxx/development/flutter
    • Framework revision 1aedbb1835 (10 days ago), 2019-10-17 08:37:27 -0700
    • Engine revision b863200c37
    • Dart version 2.5.0

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.1)
    • Android SDK at /Users/xxxxxx/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-29, build-tools 29.0.1
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.1, Build version 11A1027
    • CocoaPods version 1.8.4

[✓] Android Studio (version 3.5)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 40.2.2
    • Dart plugin version 191.8593
    • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)

[✓] VS Code (version 1.39.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.5.1

[✓] Connected device (2 available)
    • Android SDK built for x86 • emulator-5554                        • android-x86 • Android 10 (API 29) (emulator)
    • iPhone 11                 • EB26A332-279C-4282-81FF-E2B5DD96AA94 • ios         • com.apple.CoreSimulator.SimRuntime.iOS-13-1 (simulator)

• No issues found!

[BondarenkoStas]

cc @collinjackson

[robyroad]

I've the same problem with android (Galaxy Tablet A6). If i use the tablet in portrait mode it's all fine, the boxes are show in the right position around the text. When i use the tablet in landscape mode i see the same behaviour of the second image above, all the boxes are not in right place around the text.

[nachtmaar]

@BondarenkoStas @kroikie could you reproduce with the given code? any news?

[nachtmaar]

I updated the dependencies and checked again on iOS. Unfortunately, the bug still exists.

[nachtmaar]

I had another look at the problem. It works if the input image width and height are both smaller than or equal to 1000px. My guess is that the image gets resized above a certain threshold. However, it is unclear to me who does the resizing. The firebase sdk or the Google Vision API.

When using the original image with the Vision API via the demo formular the vertices are drawn correctly.

When using the max dimension of 1000 px everything works:

When leaving the image in its original resolution the vertices are incorrect:

However, the vertices are never totally wrong. It is just a scaling factor which seems to be missing. That's why I think some resizing is going on ...

[Salakar]

Given the age and inactivity of this issue I'm going to close this issue in favour of trying the latest release. Also, given the upcoming potential refactor of ML Vision to use the new Google ML Kit sdks as well I'm not sure where the best place for this issue is - as far as I know FlutterFire itself doesn't do any image resizing like mentioned above.

[nachtmaar]

@Salakar let me know then the refactoring is done. I will give it a try :)