BarcodeScanner sometimes returns old result while no barcodes are in CameraPreview

[SoundConception]

Issue:

During repeated use the BarcodeScanner sometimes returns the last previously scanned barcode result while no barcodes are in CameraPreview. I have raised this issue on stackoverflow, but have received no practical responses to date.

At around 22 seconds into this Screen Recording you can see there is no barcode in the camera preview but the previous barcode result is then quickly shown as detected.

To Reproduce:

Minimal Test App:

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'BarcodeScanner Issue',
      theme: ThemeData(primarySwatch: Colors.blue),
      home: const HomePage(),
    );
  }
}

class HomePage extends StatelessWidget {
  const HomePage({super.key});

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('Home')),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            ElevatedButton(
              onPressed: () {
                Navigator.push(
                  context,
                  MaterialPageRoute(
                    builder: (context) => const BarcodeScanPage(),
                  ),
                );
              },
              child: const Text('Scan Barcode'),
            ),
          ],
        ),
      ),
    );
  }
}

class BarcodeScanPage extends StatefulWidget {
  const BarcodeScanPage({super.key});

  @override
  State<BarcodeScanPage> createState() => BarcodeScanPageState();
}

class BarcodeScanPageState extends State<BarcodeScanPage> {
  /// Used to control the cameras on the device.
  late CameraController _controller;

  /// The current selected camera to use.
  late CameraDescription _currentCamera;

  /// The scanned [Barcode] if any.
  Barcode? _barcode;

  /// Indicates if the async initialization is complete.
  bool _initializing = true;

  /// Indicates if an [InputImage] is currently being processed.
  bool _isBusy = false;

  /// ML-Kit Barcode Scanner.
  final BarcodeScanner _barcodeScanner = BarcodeScanner();

  @override
  void initState() {
    super.initState();
    debugPrint('BarcodeScanPageState: initState()');
    _initCamera();
  }

  @override
  void dispose() {
    debugPrint('BarcodeScanPageState: dispose()');
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('Barcode Scan')),
      body: Center(
        child: _initializing
            ? const Text('Initializing...')
            : _barcode == null
                ? CameraPreview(_controller)
                : Column(
                    mainAxisSize: MainAxisSize.min,
                    children: [
                      const Text('Barcode detected'),
                      const SizedBox(height: 4.0),
                      Text(_barcode!.displayValue ?? 'Unknown'),
                    ],
                  ),
      ),
    );
  }

  /// Initialize camera and controller, and start image stream.
  Future _initCamera() async {
    final cameras = await availableCameras();
    _currentCamera = cameras.firstWhereOrNull((camera) {
          return camera.lensDirection == CameraLensDirection.back &&
              camera.sensorOrientation == 90;
        }) ??
        cameras.firstWhere((camera) {
          return camera.lensDirection == CameraLensDirection.back;
        });

    _controller = CameraController(
      _currentCamera,
      ResolutionPreset.high,
      enableAudio: false,
    );
    await _controller.initialize();
    await _controller.startImageStream(_processCameraImage);

    setState(() => _initializing = false);
  }

  /// Process a [CameraImage] into an [InputImage].
  Future _processCameraImage(CameraImage image) async {
    final WriteBuffer allBytes = WriteBuffer();
    for (final Plane plane in image.planes) {
      allBytes.putUint8List(plane.bytes);
    }
    final bytes = allBytes.done().buffer.asUint8List();

    final Size imageSize = Size(
      image.width.toDouble(),
      image.height.toDouble(),
    );

    final imageRotation = InputImageRotationValue.fromRawValue(
      _currentCamera.sensorOrientation,
    );
    if (imageRotation == null) return;

    final inputImageFormat = InputImageFormatValue.fromRawValue(
      image.format.raw,
    );
    if (inputImageFormat == null) return;

    final planeData = image.planes.map(
      (Plane plane) {
        return InputImagePlaneMetadata(
          bytesPerRow: plane.bytesPerRow,
          height: plane.height,
          width: plane.width,
        );
      },
    ).toList();

    final inputImageData = InputImageData(
      size: imageSize,
      imageRotation: imageRotation,
      inputImageFormat: inputImageFormat,
      planeData: planeData,
    );

    final inputImage = InputImage.fromBytes(
      bytes: bytes,
      inputImageData: inputImageData,
    );

    _processInputImage(inputImage);
  }

  /// Process [InputImage] for [Barcode].
  Future _processInputImage(InputImage inputImage) async {
    if (_isBusy) return;
    _isBusy = true;
    final barcodes = await _barcodeScanner.processImage(inputImage);
    if (barcodes.isNotEmpty) {
      await _controller.stopImageStream();
      await _controller.dispose();
      setState(() => _barcode = barcodes.first);
    }

    _isBusy = false;
    if (mounted) {
      setState(() {});
    }
  }
}

pubspec.yaml dependencies used:

  camera: ^0.10.0+1
  collection: ^1.16.0
  google_mlkit_barcode_scanning: ^0.4.0

Physical Testing Procedure:

1. Launch app.
2. Point camera AWAY from any potential barcodes.
3. Press Scan Barcode button => app navigates to BarcodeScanPage and starts image stream
4. Point camera AT barcode => barcode is detected and image stream is stopped
5. Navigate back to Home page.
6. Point camera AWAY from any potential barcodes again.
7. Return to step 3 and repeat until issue is observed.

Observations:

• On my old Pixel 2, the issue regularly occurs on every second visit to the BarcodeScanPage.
• On a newer Samsung Galaxy s21 it occurs after around a dozen or so visits to the BarcodeScanPage.

Debugging Attempts:

Clear the image cache:

I have tried clearing the image cache between visits to the BarcodeScanPage with the following code, but it does not change the result:

    imageCache.clearLiveImages();
    imageCache.clear();

Check if same CameraImage is being served from stream:

I have also tried adding a global previousCameraImage variable and compared the current image byte data to the previous image byte data to see if the same image is being served, by making the following code additions:

...
CameraImage? previousCameraImage;

class BarcodeScanPage extends StatefulWidget {
...
Future _processCameraImage(CameraImage image) async {
    final isPreviousCameraImage =
        image.planes.firstWhereIndexedOrNull((index, plane) {
              return !listEquals(
                plane.bytes,
                previousCameraImage?.planes[index].bytes,
              );
            }) ==
            null;
    debugPrint(
      '_processCameraImage isPreviousCameraImage: $isPreviousCameraImage',
    );
    if (isPreviousCameraImage) return;

    final WriteBuffer allBytes = WriteBuffer();
    ...

NOTE: The CameraImage class does not implement an equality operator, so I believe the above is a reasonable alternative, but am happy to be corrected!

When I run the test with this code change isPreviousCameraImage never resolves to true and the main issue still occurs.

This result, combined with the fact that the Screen Recording shows that the camera is not pointing at a barcode when the second "barcode detection" occurs, would suggest that the issue may be with the google_mlkit_barcode_scanning package.

[SoundConception]

It appears slower devices are more sensitive to this issue. Perhaps this suggests some sort of race condition is occuring?

[SoundConception]

Have confirmed the actual issue is the CameraController CameraImage stream, so will close the issue.

Determined this by converting the supplied YUV420 format CameraImage to a png each time BarcodeScanner detected a barcode, and displayed the image used next the the result. Though you don't see it in the video recording, it looks like the very first frame delivered from the CameraController CameraImage stream when it is started, is often the same as the last frame delivered before the last time the stream was stopped. i.e. the stream is not always being cleared when it is restarted.

In case anyone else ever raises a similar issue: _Always discarding the first frame delivered when calling _controller.startImageStream() solves the issue._

[zambetpentru]

@SoundConception - Thanks for sharing your observations!