[firebase_ml_vision] Only works on ResolutionPreset.high CameraController

[CoreyCole]

Describe the bug I'm building an android app using flutter that has a QR code scanning feature. I've broken out the QR code logic into a minimum reproduction case here.

Oddly enough, the QR scanning only works on my Samsung devices. My Google Pixel XL and Oneplus 6 both do not detect anything when scanning a QR code using google/firebase ml vision barcode scanning model.

To Reproduce Steps to reproduce the behavior

1. clone/build my minimum repro repo
2. point the app's camera preview at a QR code

Key code locations in the repository are:

android/app/build.gradle where I include the barcode model api:

api 'com.google.firebase:firebase-ml-vision-barcode-model:16.0.2'

lib/src/bloc/services/qr_service.dart where I run the barcode detection on the image:

_processImage(CameraImage image) async {
    if (!_alreadyCheckingImage && !_foundRoomId) {
        _alreadyCheckingImage = true;
        try {
            final barcodes = await barcodeDetector.detectInImage(
                FirebaseVisionImage.fromBytes(
                    _concatenatePlanes(image.planes),
                    FirebaseVisionImageMetadata(
                        rawFormat: image.format.raw,
                        size: Size(image.width.toDouble(), image.height.toDouble()),
                        rotation: ImageRotation.rotation0,
                        planeData: image.planes
                            .map((plane) => FirebaseVisionImagePlaneMetadata(
                                    bytesPerRow: plane.bytesPerRow,
                                    height: plane.height,
                                    width: plane.width,
                                ),
                            )
                            .toList(),
                    ),
                ),
            );
            if (barcodes != null && barcodes.length > 0) {
                try {
                    print('\n~~~');
                    print(barcodes.first.toString());
                    print(barcodes.first.displayValue);
                    print(barcodes.first.valueType);
                    print(barcodes.first.rawValue);
                    print('~~~\n');
                    final barcode = barcodes.first;
                    print(barcode.rawValue);
                    qrResult.sink.add(barcode.rawValue);
                    _foundRoomId = true;
                } catch (err, stack) {
                    print('$err\n$stack');
                }
            }
        } catch (err, stack) {
            debugPrint('$err, $stack');
        }
        _alreadyCheckingImage = false;
    }
}

Uint8List _concatenatePlanes(List<Plane> planes) {
    final WriteBuffer allBytes = WriteBuffer();
    planes.forEach((plane) => allBytes.putUint8List(plane.bytes));
    return allBytes.done().buffer.asUint8List();
}

I'm hoping I'm doing something wrong. But it is very weird that this minimum repro works flawlessly on my Samsung S8, Samsung J7, and Samsung S10+ while it does not work on my Oneplus 6 (android 10) and it does not work on my Google Pixel XL (android 9).

Additional context Dependency Versions

firebase_core: ^0.4.2+1
firebase_ml_vision: 0.9.3+3
camera: 0.5.7
rxdart: 0.22.4
device_info: 0.4.1

Device List

SM J700H      • 52033d7fe8448323   • android-arm   • Android 6.0.1 (API 23)
SM G950U1     • 988939444c58415246 • android-arm64 • Android 9 (API 28)
SM G975F      • R58M32WE95N        • android-arm64 • Android 9 (API 28)
Pixel XL      • HT76E0201647       • android-arm64 • Android 9 (API 28)
ONEPLUS A6000 • 4241cff9           • android-arm64 • Android 10 (API 29)

Working on these devices

# samsung J7
I/flutter (21900): {
I/flutter (21900):   "board": "universal7580",
I/flutter (21900):   "bootloader": "J700HXXS3BRL3",
I/flutter (21900):   "brand": "samsung",
I/flutter (21900):   "device": "j7e3g",
I/flutter (21900):   "hardware": "samsungexynos7580",
I/flutter (21900):   "host": "SWDH4623",
I/flutter (21900):   "isPhysicalDevice": true,
I/flutter (21900):   "manufacturer": "samsung",
I/flutter (21900):   "model": "SM-J700H",
I/flutter (21900):   "product": "j7e3gxx",
I/flutter (21900):   "tags": "release-keys",
I/flutter (21900):   "type": "user",
I/flutter (21900):   "version": "6.0.1",
I/flutter (21900):   "sdk": 23
I/flutter (21900): }

# samsung S8
I/flutter (22297): {
I/flutter (22297):   "board": "msm8998",
I/flutter (22297):   "bootloader": "G950U1UEU5DSC1",
I/flutter (22297):   "brand": "samsung",
I/flutter (22297):   "device": "dreamqlteue",
I/flutter (22297):   "hardware": "qcom",
I/flutter (22297):   "host": "SWDH7919",
I/flutter (22297):   "isPhysicalDevice": true,
I/flutter (22297):   "manufacturer": "samsung",
I/flutter (22297):   "model": "SM-G950U1",
I/flutter (22297):   "product": "dreamqlteue",
I/flutter (22297):   "tags": "release-keys",
I/flutter (22297):   "type": "user",
I/flutter (22297):   "version": "9",
I/flutter (22297):   "sdk": 28
I/flutter (22297): }

# samsung S10+
I/flutter (10037): {
I/flutter (10037):   "board": "universal9820",
I/flutter (10037):   "bootloader": "G975FXXS3ASJG",
I/flutter (10037):   "brand": "samsung",
I/flutter (10037):   "device": "beyond2",
I/flutter (10037):   "hardware": "exynos9820",
I/flutter (10037):   "host": "SWDG4719",
I/flutter (10037):   "isPhysicalDevice": true,
I/flutter (10037):   "manufacturer": "samsung",
I/flutter (10037):   "model": "SM-G975F",
I/flutter (10037):   "product": "beyond2ltexx",
I/flutter (10037):   "tags": "release-keys",
I/flutter (10037):   "type": "user",
I/flutter (10037):   "version": "9",
I/flutter (10037):   "sdk": 28
I/flutter (10037): }

Not working on these devices

# pixel XL
I/flutter ( 1500): {
I/flutter ( 1500):   "board": "marlin",
I/flutter ( 1500):   "bootloader": "8996-012001-1904111134",
I/flutter ( 1500):   "brand": "google",
I/flutter ( 1500):   "device": "marlin",
I/flutter ( 1500):   "hardware": "marlin",
I/flutter ( 1500):   "host": "abfarm826",
I/flutter ( 1500):   "isPhysicalDevice": true,
I/flutter ( 1500):   "manufacturer": "Google",
I/flutter ( 1500):   "model": "Pixel XL",
I/flutter ( 1500):   "product": "marlin",
I/flutter ( 1500):   "tags": "release-keys",
I/flutter ( 1500):   "type": "user",
I/flutter ( 1500):   "version": "9",
I/flutter ( 1500):   "sdk": 28
I/flutter ( 1500): }

# oneplus 6
I/flutter (15401): {
I/flutter (15401):   "board": "sdm845",
I/flutter (15401):   "bootloader": "unknown",
I/flutter (15401):   "brand": "OnePlus",
I/flutter (15401):   "device": "OnePlus6",
I/flutter (15401):   "hardware": "qcom",
I/flutter (15401):   "host": "rd-build-73",
I/flutter (15401):   "isPhysicalDevice": true,
I/flutter (15401):   "manufacturer": "OnePlus",
I/flutter (15401):   "model": "ONEPLUS A6000",
I/flutter (15401):   "product": "OnePlus6",
I/flutter (15401):   "tags": "release-keys",
I/flutter (15401):   "type": "user",
I/flutter (15401):   "version": "10",
I/flutter (15401):   "sdk": 29
I/flutter (15401): }

flutter doctor -v

[✓] Flutter (Channel stable, v1.9.1+hotfix.5, on Mac OS X 10.14.6 18G95, locale en-US)
    • Flutter version 1.9.1+hotfix.5 at /Users/corey/flutter
    • Framework revision 1aedbb1835 (6 weeks ago), 2019-10-17 08:37:27 -0700
    • Engine revision b863200c37
    • Dart version 2.5.0

[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at /Users/corey/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-28, build-tools 28.0.3
    • ANDROID_HOME = /Users/corey/Library/Android/sdk
    • ANDROID_SDK_ROOT = /Users/corey/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.2.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.2.1, Build version 11B500
    • CocoaPods version 1.7.5

[✓] Android Studio (version 3.5)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 40.2.2
    • Dart plugin version 191.8593
    • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)

[✓] VS Code (version 1.40.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.6.0

[✓] Connected device (4 available)
    • ONEPLUS A6000 • 4241cff9           • android-arm64 • Android 10 (API 29)
    • SM J700H      • 52033d7fe8448323   • android-arm   • Android 6.0.1 (API 23)
    • SM G950U1     • 988939444c58415246 • android-arm64 • Android 9 (API 28)
    • Pixel XL      • HT76E0201647       • android-arm64 • Android 9 (API 28)

• No issues found!

Posted on Stack Overflow as well in case I'm doing something wrong and this isn't actually a bug with firebase ML vision

[CoreyCole]

We fixed the issue by changing the ResolutionPreset on the CameraController to high:

final newCameraController = CameraController(
    cameras.first,
    ResolutionPreset.high,
    enableAudio: false,
);

We are only using the camera preview for scanning a QR code, so high resolution is overkill for our use case. On lower-end devices it causes dropped frames to set the camera controller to high. Would love to be able to use this plugin on lower resolution presets.

[eliasjtg]

Same issue, high resolution is overkill, medium crash

[rmtmckenzie]

I'm seeing the same thing with a Nexus 5x, on medium it doesn't find anything.

[CoreyCole]

For anyone else looking only to scan QR codes, we decided to use qrcode. It uses native packages AVCaptureSession in iOS and zxing in Android.

https://pub.dev/packages/qrcode

Running the firebase_ml_vision for QR codes was overkill, even more so with the resolution pinned at high.

[claireliu14]

I met the issue when I run app on pixel 3a with camera streaming and ml_vision. If ResolutionPresent is setting lower than high, ml_vision will detect nothing.

But this problem not appear when I debug on Samsung s8 and iPhone 6s. They can only work when I set ResolutionPresent to medium. CameraPreview will lag when ResolutionPresent set high.

Any solution ?

[sgehrman]

Same issue, the example app doesn't even work on Android.

[greenlihui]

Similar thing also happened here, I am writing a real time face detection app using face detector, and I was setting ResolutionPreset to medium, which results no faces detected at all, it took me days to find out the reason. My device is Samsung S9 running Android X(api 29), face detection working fine if preset is not set to medium.

[topnax]

I can relate to this. Text OCR works on ResolutionPreset.HIGH, but on slower devices it might stutter a bit. On Samsung S8 it stutters hell a lot and is completely unusable. Switching to ResolutionPreset.MEDIUM fixes the Samsung S8 issue, but completely disables other devices as they no longer recognize a single character from the image.

[benjastudio]

Same Problem here. The medium resolution was not working at all.

I looked closer at what happened, and I found that my phone (redmi 4X) added a padding to each row of the planes when the medium resolution was selected (but not with high or low resolution).

In my case the image format is a YUV 4:2:0, meaning the image is decomposed in 3 planes (Y, U, V), with 8 bits per pixel for Y, and 4 for U and V.

Each plane is a matrix of ~pixel stored in a 1D-array. They come with a bytesPerRow value that represents the length of a row. This may be or not the same size than the image width depending if some padding is added at the end of each row or not. On my case, bytesPerRow was 768and my image width 720 (so a padding of 48 bytes has been added to each row).

As bytesPerRow is given to the FirebaseVisionImagePlaneMetadata object I suppose that the row stride should be handled correctly by the library. But it looks that something is not handled right?

1 - I can't help and dig more into this than I actually did.

2 - I wrote 2 workarounds (A and B) that keep the data consistency:

• A: Remove manually the padding in the plane (for each row) and set the bytesPerRow to the image width. Drawback: It's a bit painful and I suppose it may slow down a bit the process.

• B: Change the image width to include the padding. Drawback: your image has a "black border" on the right of the image of the size of the padding. It may not be a problem in some cases.

// For each plane
FirebaseVisionImagePlaneMetadata(
        bytesPerRow: plane.bytesPerRow,
        width: plane.bytesPerRow,
        height: image.height,
      )

...

  int newWidth = planes[0].bytesPerRow;
  // Build image metadata
  var metadata = FirebaseVisionImageMetadata(
    rawFormat: image.format.raw,
    size: Size(newWidth.toDouble(), image.height.toDouble()),
    rotation: ImageRotation.rotation90,
    planeData: planesMetadata,
  );
...

Both are working for my own but they still are workaround. I hope this will help some of you.

[SubashManian]

Bar Code scanner won't run iPhone 8 device while set it max resolution. if i set resolution to medium at the time it work's fine but can't able to scan the lower quality bar code in printed copies. while set to max resolution iPhone 6, 6s, 7, 7 plus, 8, 8 plus memory consumption is more than gb and it leads to crash. Attached Memory usage screen shot. please advice

[decolon]

I have the same problem on my Pixel 3a. Only works with max resolution, and in max resolution the memory builds until the app crashes.

[AHuminskyi]

We fixed the issue by changing the ResolutionPreset on the CameraController to high:

final newCameraController = CameraController(
    cameras.first,
    ResolutionPreset.high,
    enableAudio: false,
);

We are only using the camera preview for scanning a QR code, so high resolution is overkill for our use case. On lower-end devices it causes dropped frames to set the camera controller to high. Would love to be able to use this plugin on lower resolution presets.

ResolutionPreset.high or even ultra didn't help me on OnePlus 6t :(

[IliaKhuzhakhmetovRoonyx]

whats new?

[svenjacobs]

I'm having the same problem on a Google Pixel (1) and Pixel 4 XL. Only with ResolutionPreset.high are barcodes detected reliably.

[TheArchitect123]

whats new?

nothing, by the looks of it. This issue has been opened for over a year now, so I'm assuming that nobody's looking at this. If I were you I would be looking at using another library