App Crashes on Physical Device During Continuous Camera Stream for Real-Time Face Detection in Flutter

[EsioFreitas]

Describe your issue. If applicable, add screenshots to help explain your problem.

I'm facing an issue in my Flutter app related to using camera streams for real-time face detection. The app works fine on the emulator, but on a physical device, it freezes and eventually crashes when handling the continuous image stream from the camera. I've already configured camera orientation lock and added delays to avoid overload, but the problem persists. Has anyone encountered this issue before and could help me identify and resolve the cause of these crashes when dealing with camera streams on a real device?

Code sample

original code

```dart import 'dart:async'; import 'dart:io'; import 'dart:math'; import 'package:camera/camera.dart'; import 'package:flutter/foundation.dart'; import 'package:flutter/material.dart'; import 'package:flutter/services.dart'; import 'package:flutter_easyloading/flutter_easyloading.dart'; import 'package:fura_fila/utils/colors.dart'; import 'package:fura_fila/views/widgets/base.dart'; import 'package:google_mlkit_face_detection/google_mlkit_face_detection.dart'; class FaceDetectionScreen extends StatefulWidget { final List cameras; const FaceDetectionScreen({ Key? key, required this.cameras, }) : super(key: key); @override _FaceDetectionScreenState createState() => _FaceDetectionScreenState(); } class _FaceDetectionScreenState extends State with WidgetsBindingObserver { late CameraController _cameraController; late Future _initializeControllerFuture; late final FaceDetector _faceDetector; bool _isDetecting = false; int _detectingCounter = 0; String _feedbackMessage = 'Posicione seu rosto na câmera'; final Duration _detectionInterval = const Duration(milliseconds: 500); DateTime _lastDetectionTime = DateTime.now(); @override void initState() { super.initState(); _faceDetector = FaceDetector( options: FaceDetectorOptions( enableClassification: true, minFaceSize: 0.1, ), ); WidgetsBinding.instance.addObserver(this); _initializeControllerFuture = _initializeCamera(); } @override void dispose() { WidgetsBinding.instance.removeObserver(this); _cameraController.dispose(); _faceDetector.close(); super.dispose(); } Future _initializeCamera() async { try { _cameraController = CameraController( widget.cameras.isNotEmpty ? widget.cameras[1] : widget.cameras.first, ResolutionPreset.low, enableAudio: false, ); await _cameraController.initialize(); await _cameraController .lockCaptureOrientation(DeviceOrientation.portraitUp); if (_cameraController.value.isInitialized) { _cameraController.startImageStream((CameraImage image) { if (!_isDetecting) { _isDetecting = true; _detectFaces(image); } }); } setState(() {}); } catch (e) { print('Erro ao inicializar a câmera: $e'); } } Future _detectFaces(CameraImage image) async { if (_isDetecting || DateTime.now().difference(_lastDetectionTime) < _detectionInterval) { return; } _isDetecting = true; _lastDetectionTime = DateTime.now(); try { final inputImage = _convertCameraImageToInputImage(image); final List faces = await _faceDetector.processImage(inputImage); setState(() { if (faces.isNotEmpty) { final face = faces[0]; if (!_isBackgroundWhite(image)) { _feedbackMessage = 'Certifique-se de que o fundo é branco.'; return; } if (_isFaceTooCloseOrFar(face)) { _feedbackMessage = 'Por favor, ajuste a distância do rosto.'; return; } if (_isFaceCovered(face)) { _feedbackMessage = 'Certifique-se de que o rosto não está coberto.'; return; } if (_isHeadCovered(face)) { _feedbackMessage = 'Algo está cobrindo a cabeça.'; return; } _feedbackMessage = 'Rosto detectado corretamente.'; } else { _feedbackMessage = 'Nenhum rosto detectado.'; } }); } catch (e) { print('Erro ao detectar rostos: $e'); } finally { _isDetecting = false; } } bool _isFaceTooCloseOrFar(Face face) { final faceWidth = face.boundingBox.width; if (faceWidth < 100 || faceWidth > 300) { return true; } return false; } bool _isBackgroundWhite(CameraImage image) { int whitePixelCount = 0; int totalPixelCount = 0; for (int i = 0; i < image.planes[0].bytes.length; i += 4) { final r = image.planes[0].bytes[i]; final g = image.planes[0].bytes[i + 1]; final b = image.planes[0].bytes[i + 2]; if (r > 200 && g > 200 && b > 200) { whitePixelCount++; } totalPixelCount++; } double whitePercentage = (whitePixelCount / totalPixelCount) * 100; return whitePercentage > 50; } bool _isFaceCovered(Face face) { final leftEye = face.landmarks[FaceLandmarkType.leftEye]; final rightEye = face.landmarks[FaceLandmarkType.rightEye]; final noseBase = face.landmarks[FaceLandmarkType.noseBase]; if (leftEye == null || rightEye == null || noseBase == null) { return true; } return false; } bool _isHeadCovered(Face face) { final forehead = face.landmarks[FaceLandmarkType.noseBase]; if (forehead == null) { return true; } return false; } InputImage _convertCameraImageToInputImage(CameraImage image) { final WriteBuffer allBytes = WriteBuffer(); for (final Plane plane in image.planes) { allBytes.putUint8List(plane.bytes); } final bytes = allBytes.done().buffer.asUint8List(); InputImageRotation rotation; if (_cameraController.description.lensDirection == CameraLensDirection.front) { rotation = InputImageRotation.rotation270deg; } else { rotation = InputImageRotation.rotation90deg; } final inputImage = InputImage.fromBytes( bytes: bytes, metadata: InputImageMetadata( size: Size(image.width.toDouble(), image.height.toDouble()), rotation: rotation, format: InputImageFormat.nv21, bytesPerRow: image.planes.first.bytesPerRow, ), ); return inputImage; } @override Widget build(BuildContext context) { return Base( bottomButton: ElevatedButton( style: ElevatedButton.styleFrom( backgroundColor: AppColors.brownDark, padding: const EdgeInsets.symmetric(vertical: 16), shape: RoundedRectangleBorder( borderRadius: BorderRadius.circular(8), ), ), onPressed: () async { try { final dialogContext = context; await _initializeControllerFuture; final image = await _cameraController.takePicture(); if (!mounted) return; showDialog( context: dialogContext, builder: (BuildContext context) { return AlertDialog( shape: RoundedRectangleBorder( borderRadius: BorderRadius.circular(10), ), title: const Text('Foto Capturada'), content: Column( mainAxisSize: MainAxisSize.min, children: [ Image.file(File(image.path)), const SizedBox(height: 20), const Text('Deseja enviar a foto ou cancelar?'), ], ), actions: [ TextButton( onPressed: () { Navigator.of(dialogContext).pop(); }, child: const Text('Cancelar'), ), TextButton( onPressed: () {}, child: const Text('Enviar'), ), ], ); }, ); } catch (e) { print('Erro ao capturar a foto: $e'); } }, child: const Text( 'SALVAR FOTO', style: TextStyle( color: Colors.white, fontWeight: FontWeight.bold, fontSize: 18, ), ), ), children: [ Padding( padding: const EdgeInsets.symmetric(vertical: 1), child: Row( mainAxisAlignment: MainAxisAlignment.start, children: [ IconButton( icon: const Icon(Icons.arrow_back), onPressed: () { Navigator.of(context).pop(); }, ), Text( _feedbackMessage, textAlign: TextAlign.center, style: const TextStyle( fontWeight: FontWeight.bold, fontSize: 20, ), ), ], ), ), const SizedBox(height: 20), Expanded( child: FutureBuilder( future: _initializeControllerFuture, builder: (context, snapshot) { if (snapshot.connectionState == ConnectionState.done) { if (snapshot.hasError) { return const Center(child: Text('Erro ao iniciar a câmera')); } else { return CameraPreview(_cameraController); } } else { return const Center(child: CircularProgressIndicator()); } }, ), ), const SizedBox(height: 20), ], ); } } ```

code to facilitate testing

```dart import 'dart:async'; import 'dart:io'; import 'package:camera/camera.dart'; import 'package:flutter/material.dart'; import 'package:flutter/services.dart'; import 'package:google_mlkit_face_detection/google_mlkit_face_detection.dart'; void main() async { WidgetsFlutterBinding.ensureInitialized(); final cameras = await availableCameras(); runApp(MyApp(cameras: cameras)); } class MyApp extends StatelessWidget { final List cameras; const MyApp({Key? key, required this.cameras}) : super(key: key); @override Widget build(BuildContext context) {    return MaterialApp(      title: 'Face Detection App',      theme: ThemeData(        primarySwatch: Colors.blue,      ),      home: FaceDetectionScreen(cameras: cameras),    ); } } class FaceDetectionScreen extends StatefulWidget { final List cameras; const FaceDetectionScreen({    Key? key,    required this.cameras, }) : super(key: key); @override _FaceDetectionScreenState createState() => _FaceDetectionScreenState(); } class _FaceDetectionScreenState extends State { late CameraController _cameraController; late Future _initializeControllerFuture; late final FaceDetector _faceDetector; bool _isDetecting = false; String _feedbackMessage = 'Posicione seu rosto na câmera'; @override void initState() {    super.initState();    _faceDetector = FaceDetector(      options: FaceDetectorOptions(        enableClassification: true,        minFaceSize: 0.1,      ),    );    _initializeControllerFuture = _initializeCamera(); } @override void dispose() {    _cameraController.dispose();    _faceDetector.close();    super.dispose(); } Future _initializeCamera() async {    try {      _cameraController = CameraController(        widget.cameras.isNotEmpty ? widget.cameras[1] : widget.cameras.first,        ResolutionPreset.low,        enableAudio: false,      );      await _cameraController.initialize();      await _cameraController          .lockCaptureOrientation(DeviceOrientation.portraitUp);      _cameraController.startImageStream((CameraImage image) {        if (!_isDetecting) {          _isDetecting = true;          _detectFaces(image);        }      });      setState(() {});    } catch (e) {      print('Erro ao inicializar a câmera: $e');    } } Future _detectFaces(CameraImage image) async {    final inputImage = _convertCameraImageToInputImage(image);    final List faces = await _faceDetector.processImage(inputImage);    setState(() {      if (faces.isNotEmpty) {        _feedbackMessage = 'Rosto detectado corretamente.';      } else {        _feedbackMessage = 'Nenhum rosto detectado.';      }    });    _isDetecting = false; } InputImage _convertCameraImageToInputImage(CameraImage image) {    final bytes = image.planes.first.bytes;    return InputImage.fromBytes(      bytes: bytes,      metadata: InputImageMetadata(        size: Size(image.width.toDouble(), image.height.toDouble()),        rotation: InputImageRotation.rotation0deg,        format: InputImageFormat.nv21,        bytesPerRow: image.planes.first.bytesPerRow,      ),    ); } @override Widget build(BuildContext context) {    return Scaffold(      appBar: AppBar(        title: Text(_feedbackMessage),      ),      body: FutureBuilder(        future: _initializeControllerFuture,        builder: (context, snapshot) {          if (snapshot.connectionState == ConnectionState.done) {            if (snapshot.hasError) {              return Center(child: Text('Erro ao iniciar a câmera'));            } else {              return CameraPreview(_cameraController);            }          } else {            return Center(child: CircularProgressIndicator());          }        },      ),      floatingActionButton: FloatingActionButton(        onPressed: () async {          try {            await _initializeControllerFuture;            final image = await _cameraController.takePicture();            if (!mounted) return;            showDialog(              context: context,              builder: (BuildContext context) {                return AlertDialog(                  shape: RoundedRectangleBorder(                    borderRadius: BorderRadius.circular(10),                  ),                  title: const Text('Foto Capturada'),                  content: Column(                    mainAxisSize: MainAxisSize.min,                    children: [                      Image.file(File(image.path)),                      const SizedBox(height: 20),                      const Text('Deseja enviar a foto ou cancelar?'),                    ],                  ),                  actions: [                    TextButton(                      onPressed: () {                        Navigator.of(context).pop();                      },                      child: const Text('Cancelar'),                    ),                    TextButton(                      onPressed: () {                        // Logica para envio da foto                      },                      child: const Text('Enviar'),                    ),                  ],                );              },            );          } catch (e) {            print('Erro ao capturar a foto: $e');          }        },        child: const Icon(Icons.camera_alt),      ),    ); } } ```

Screenshots or Video

Screenshots / Video demonstration


Logs

Logs

```console [Paste your logs here] ```

Flutter Doctor output

Doctor output

```console [✓] Flutter (Channel stable, 3.24.3, on macOS 14.5 23F79 darwin-arm64, locale pt-BR) • Flutter version 3.24.3 on channel stable at /Users/esiogustavopereirafreitas/development/flutter • Upstream repository https://github.com/flutter/flutter.git • Framework revision 2663184aa7 (6 weeks ago), 2024-09-11 16:27:48 -0500 • Engine revision 36335019a8 • Dart version 3.5.3 • DevTools version 2.37.3 [✓] Android toolchain - develop for Android devices (Android SDK version 35.0.0) • Android SDK at /Users/esiogustavopereirafreitas/Library/Android/sdk • Platform android-35, build-tools 35.0.0 • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 17.0.11+0-17.0.11b1207.24-11852314) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS (Xcode 15.4) • Xcode at /Applications/Xcode.app/Contents/Developer • Build 15F31d • CocoaPods version 1.15.2 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] Android Studio (version 2024.1) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 17.0.11+0-17.0.11b1207.24-11852314) [✓] VS Code (version 1.91.1) • VS Code at /Users/esiogustavopereirafreitas/Downloads/Visual Studio Code.app/Contents • Flutter extension version 3.98.0 [✓] Connected device (4 available) • sdk gphone64 arm64 (mobile) • emulator-5554 • android-arm64 • Android 15 (API 35) (emulator) • macOS (desktop) • macos • darwin-arm64 • macOS 14.5 23F79 darwin-arm64 • Mac Designed for iPad (desktop) • mac-designed-for-ipad • darwin • macOS 14.5 23F79 darwin-arm64 • Chrome (web) • chrome • web-javascript • Google Chrome 129.0.6668.103 ! Error: Browsing on the local area network for iPhone. Ensure the device is unlocked and attached with a cable or associated with the same local area network as this Mac. The device must be opted into Developer Mode to connect wirelessly. (code -27) [✓] Network resources • All expected network resources are available. • No issues found! ```

Steps to reproduce.

Steps to reproduce 1- Set up a Flutter project with camera and face detection functionality using camera and google_mlkit_face_detection packages. 2- Initialize the camera and start the image stream for real-time face detection. 3- Run the app on a physical device (the issue does not occur on the emulator). 4- Attempt to detect faces using the continuous camera stream. 5- The app may crash when handling the camera's continuous image stream.

What is the expected result?

Expected results

1- The app should run smoothly on a physical device, continuously detecting faces from the camera stream without crashing. 2- The camera stream should provide real-time data for face detection without causing any performance issues.

Actual results

1- The app works fine on an emulator, but on a physical device, it freezes and crashes when handling the continuous camera stream. 2- The app crashes even after setting delays to prevent overloading and locking the camera orientation.

Did you try our example app?

Yes

Is it reproducible in the example app?

Yes

Reproducible in which OS?

iOS and Android

Flutter/Dart Version?

Flutter 3.24.3 • channel stable • https://github.com/flutter/flutter.git Framework • revision 2663184aa7 (6 weeks ago) • 2024-09-11 16:27:48 -0500 Engine • revision 36335019a8 Tools • Dart 3.5.3 • DevTools 2.37.3

Plugin Version?

camera: ^0.11.0+2 google_mlkit_face_detection: ^0.11.0 google_mlkit_commons: ^0.8.1 camera_android_camerax: 0.6.7+2

[santhoshAndroid]

Does this face detection work on Android?

[bensonarafat]

@EsioFreitas Do you try the example app?