🐛 [firebase_ml_vision] Face detection from camera live feed stopped working on Android devices

pixnbit commented 4 years ago

Bug report

Describe the bug We built a face verification feature, with the first step detecting face rectangles using firebase_ml_vision face detector, in January. Both the iOS and Android versions tested working correctly. We recently noticed the Android app stopped detecting faces for further processing. This is a regression bug certain library update broke the face detection. Then we tested the example app in this repo, the issue can be reproduced.

Steps to reproduce

Steps to reproduce the behavior:

The problem is only on Android, not on iOS
We were able to get face detection working on Android before and it regressed recently.
After some debugging and research, I can confirm that still images from the disk still works. Only the frames grabbed from the camera feed in YUV 420 result in no faces detected.
I tested your official example camera scanner, live face detection text/label/bar on Android does not work.
Devices tested: Pixel and Galaxy Tab S5e

Expected behavior

Faces in the camera feed would have rectangles.

Additional context

Add any other context about the problem here.

Flutter doctor

Run flutter doctor and paste the output below:

Click To Expand

``` Doctor summary (to see all details, run flutter doctor -v): [✓] Flutter (Channel stable, 1.22.0, on Mac OS X 10.15.7 19H2, locale en-US) [✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3) [✓] Xcode - develop for iOS and macOS (Xcode 12.0.1) [✓] Android Studio (version 4.0) [!] IntelliJ IDEA Ultimate Edition (version 2020.2.2) ✗ Flutter plugin not installed; this adds Flutter specific functionality. ✗ Dart plugin not installed; this adds Dart specific functionality. [✓] VS Code (version 1.48.2) [✓] Connected device (1 available) ! Doctor found issues in 1 category. ```

Flutter dependencies

Run flutter pub deps -- --style=compact and paste the output below:

Click To Expand

``` Dart SDK 2.10.0 Flutter SDK 1.22.0 firebase_ml_vision_example 0.0.0 dependencies: - camera 0.5.8+7 [flutter] - cupertino_icons 0.1.3 - firebase_core 0.5.0 [firebase_core_platform_interface flutter quiver meta firebase_core_web] - firebase_ml_vision 0.9.7 [flutter] - flutter 0.0.0 [characters collection meta typed_data vector_math sky_engine] - image_picker 0.6.7+11 [flutter flutter_plugin_android_lifecycle image_picker_platform_interface] dev dependencies: - e2e 0.6.3 [flutter flutter_driver flutter_test path] - flutter_driver 0.0.0 [file json_rpc_2 meta path web_socket_channel vm_service_client webdriver flutter flutter_test fuchsia_remote_debug_protocol archive args async boolean_selector characters charcode clock collection convert crypto fake_async matcher platform process pub_semver source_span stack_trace stream_channel string_scanner sync_http term_glyph test_api typed_data vector_math] - flutter_test 0.0.0 [flutter test_api path fake_async clock stack_trace vector_math async boolean_selector characters charcode collection matcher meta source_span stream_channel string_scanner term_glyph typed_data] - path 1.8.0-nullsafety.1 - path_provider 1.6.18 [flutter path_provider_platform_interface path_provider_macos path_provider_linux path_provider_windows] - pedantic 1.10.0-nullsafety.1 - test 1.16.0-nullsafety.5 [analyzer async boolean_selector coverage http http_multi_server io js node_preamble package_config path pedantic pool shelf shelf_packages_handler shelf_static shelf_web_socket source_span stack_trace stream_channel typed_data web_socket_channel webkit_inspection_protocol yaml test_api test_core] dependency overrides: - firebase_core 0.5.0 [firebase_core_platform_interface flutter quiver meta firebase_core_web] transitive dependencies: - _fe_analyzer_shared 11.0.0 [meta] - analyzer 0.40.4 [_fe_analyzer_shared args charcode cli_util collection convert crypto glob meta package_config path pub_semver source_span watcher yaml] - archive 2.0.13 [crypto args path] - args 1.6.0 - async 2.5.0-nullsafety.1 [collection] - boolean_selector 2.1.0-nullsafety.1 [source_span string_scanner] - characters 1.1.0-nullsafety.3 - charcode 1.2.0-nullsafety.1 - cli_util 0.2.0 [path] - clock 1.1.0-nullsafety.1 - collection 1.15.0-nullsafety.3 - convert 2.1.1 [charcode typed_data] - coverage 0.14.1 [args logging package_config path source_maps stack_trace vm_service] - crypto 2.1.5 [collection convert typed_data] - fake_async 1.2.0-nullsafety.1 [clock collection] - ffi 0.1.3 - file 6.0.0-nullsafety.2 [meta path] - firebase 7.3.0 [http http_parser js] - firebase_core_platform_interface 2.0.0 [flutter meta plugin_platform_interface quiver] - firebase_core_web 0.2.0 [firebase firebase_core_platform_interface flutter flutter_web_plugins meta js] - flutter_plugin_android_lifecycle 1.0.11 [flutter] - flutter_web_plugins 0.0.0 [flutter characters collection meta typed_data vector_math] - fuchsia_remote_debug_protocol 0.0.0 [json_rpc_2 process web_socket_channel flutter_test flutter_driver archive args async boolean_selector charcode clock collection convert crypto fake_async file matcher meta path platform pub_semver source_span stack_trace stream_channel string_scanner sync_http term_glyph test_api typed_data vector_math vm_service_client webdriver] - glob 1.2.0 [async collection node_io path pedantic string_scanner] - http 0.12.2 [http_parser path pedantic] - http_multi_server 2.2.0 [async] - http_parser 3.1.4 [charcode collection source_span string_scanner typed_data] - image_picker_platform_interface 1.1.1 [flutter meta http plugin_platform_interface] - io 0.3.4 [charcode meta path string_scanner] - js 0.6.3-nullsafety.1 - json_rpc_2 2.2.2 [stack_trace stream_channel] - logging 0.11.4 - matcher 0.12.10-nullsafety.1 [stack_trace] - meta 1.3.0-nullsafety.3 - mime 0.9.7 - node_interop 1.1.1 [js] - node_io 1.1.1 [node_interop path] - node_preamble 1.4.12 - package_config 1.9.3 [path charcode] - path_provider_linux 0.0.1+2 [path xdg_directories path_provider_platform_interface flutter] - path_provider_macos 0.0.4+4 [flutter] - path_provider_platform_interface 1.0.3 [flutter meta platform plugin_platform_interface] - path_provider_windows 0.0.4+1 [path_provider_platform_interface meta path flutter ffi win32] - platform 3.0.0-nullsafety.2 - plugin_platform_interface 1.0.3 [meta] - pool 1.5.0-nullsafety.1 [async stack_trace] - process 4.0.0-nullsafety.2 [file path platform] - pub_semver 1.4.4 [collection] - quiver 2.1.3 [matcher meta] - shelf 0.7.9 [async collection http_parser path stack_trace stream_channel] - shelf_packages_handler 2.0.0 [path shelf shelf_static] - shelf_static 0.2.8 [convert http_parser mime path shelf] - shelf_web_socket 0.2.3 [shelf web_socket_channel stream_channel] - sky_engine 0.0.99 - source_map_stack_trace 2.1.0-nullsafety.2 [path stack_trace source_maps] - source_maps 0.10.10-nullsafety.1 [source_span] - source_span 1.8.0-nullsafety.2 [charcode collection path term_glyph] - stack_trace 1.10.0-nullsafety.1 [path] - stream_channel 2.1.0-nullsafety.1 [async] - string_scanner 1.1.0-nullsafety.1 [charcode source_span] - sync_http 0.2.0 - term_glyph 1.2.0-nullsafety.1 - test_api 0.2.19-nullsafety.2 [async boolean_selector collection meta path source_span stack_trace stream_channel string_scanner term_glyph matcher] - test_core 0.3.12-nullsafety.5 [analyzer async args boolean_selector collection coverage glob io meta package_config path pedantic pool source_map_stack_trace source_maps source_span stack_trace stream_channel vm_service yaml matcher test_api] - typed_data 1.3.0-nullsafety.3 [collection] - vector_math 2.1.0-nullsafety.3 - vm_service 5.2.0 [meta] - vm_service_client 0.2.6+2 [async collection json_rpc_2 pub_semver source_span stack_trace stream_channel web_socket_channel] - watcher 0.9.7+15 [async path pedantic] - web_socket_channel 1.1.0 [async crypto stream_channel] - webdriver 2.1.2 [archive matcher path stack_trace sync_http] - webkit_inspection_protocol 0.7.3 [logging] - win32 1.7.3 [ffi] - xdg_directories 0.1.2 [meta path process] - yaml 2.2.1 [charcode collection string_scanner source_span] ```

darshankawar commented 4 years ago

@pixnbit Tried the official ml_vision example on Android 8.1.0 device on latest beta and the face detection worked properly.

3798.mp4.zip

flutter doctor -v

``` [✓] Flutter (Channel beta, 1.22.0, on Mac OS X 10.15.4 19E2269, locale en-IN) • Flutter version 1.22.0 at /Users/dhs/documents/Fluttersdk/flutter • Framework revision d408d302e2 (8 days ago), 2020-09-29 11:49:17 -0700 • Engine revision 5babba6c4d • Dart version 2.10.0 [✓] Android toolchain - develop for Android devices (Android SDK version 30.0.0) • Android SDK at /Users/dhs/Library/Android/sdk • Platform android-30, build-tools 30.0.0 • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS (Xcode 12.0.1) • Xcode at /Applications/Xcode.app/Contents/Developer • Xcode 12.0.1, Build version 12A7300 • CocoaPods version 1.9.3 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] Android Studio (version 4.0) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin version 46.0.2 • Dart plugin version 193.7361 • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593) [✓] VS Code (version 1.49.1) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.11.0 [✓] Connected device (3 available) • Darshan's iphone (mobile) • 21150b119064aecc249dfcfe05e259197461ce23 • ios • iOS 14.0 • Web Server (web) • web-server • web-javascript • Flutter Tools • Chrome (web) • chrome • web-javascript • Google Chrome 86.0.4240.75 • No issues found! ```

Can you provide video of the issue you are facing ? Thanks.

pixnbit commented 4 years ago

Here are my test videos, both Android 9 and 10 are failing, iOS works.

Google Pixels (Two phones, Android 9 and 10), Samsung Galaxy Tab s5e Android 10

Samsung Galaxy Tab s5e Android 10.MP4.zip

Google Pixel Android 9.MP4.zip

This is with iPhone: iPhone X.MP4.zip

pixnbit commented 4 years ago

Any update one this?

darshankawar commented 4 years ago

@pixnbit Tried again on latest stable (1.22.1) using Samsung S10+ device and was able to detect faces properly.

3798.mp4.zip

flutter doctor -v

``` [✓] Flutter (Channel stable, 1.22.1, on Mac OS X 10.15.4 19E2269, locale en-IN) • Flutter version 1.22.1 at /Users/dhs/documents/Fluttersdk/flutter • Framework revision f30b7f4db9 (5 days ago), 2020-10-08 10:06:30 -0700 • Engine revision 75bef9f6c8 • Dart version 2.10.1 [✓] Android toolchain - develop for Android devices (Android SDK version 30.0.0) • Android SDK at /Users/dhs/Library/Android/sdk • Platform android-30, build-tools 30.0.0 • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS (Xcode 12.0.1) • Xcode at /Applications/Xcode.app/Contents/Developer • Xcode 12.0.1, Build version 12A7300 • CocoaPods version 1.9.3 [✓] Android Studio (version 4.0) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin version 46.0.2 • Dart plugin version 193.7361 • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593) [✓] VS Code (version 1.49.1) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.11.0 [✓] Connected device (1 available) • iPhone SE (2nd generation) (mobile) • 6C85835D-FBFD-4AB3-8DE8-B4FAD35E5367 • ios • com.apple.CoreSimulator.SimRuntime.iOS-14-0 (simulator) • No issues found! ```

tindalida commented 4 years ago

Hi, I am encountering the same problem for some Android devices, only barcode and face detection is not working.

I have tested with several phones and getting inconsistent results, as I have observed it is working better at lower versions of android.

Below is a testing comparison of both devices running on Android 10

Samsung S20 FE - detection not working 121983714_369105344502434_2903725326437473129_n

Samsung XCover4s - detection is ok 122206821_351353679534731_1388059525097919247_n

[√] Flutter (Channel stable, 1.22.2, on Microsoft Windows [Version 10.0.18363.1139], locale en-SG)
    • Flutter version 1.22.2 at C:\src\flutter
    • Framework revision 84f3d28555 (6 days ago), 2020-10-15 16:26:19 -0700
    • Engine revision b8752bbfff
    • Dart version 2.10.2

[√] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
    • Android SDK at C:\Android\android-sdk
    • Platform android-29, build-tools 29.0.2
    • ANDROID_HOME = C:\Android\android-sdk
    • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
    • All Android licenses accepted.

[!] Android Studio (version 4.1.0)
    • Android Studio at C:\Program Files\Android\Android Studio
    X Flutter plugin not installed; this adds Flutter specific functionality.
    X Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)

[√] VS Code (version 1.50.1)
    • VS Code at C:\Users\Christine\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.15.1

[!] Connected device
    ! No devices available

! Doctor found issues in 2 categories

(I have flutter and dart plugins in Android studio but apparently it doesn't reflect)

Any ideas about this inconsistency?

Thanks.

pixnbit commented 4 years ago

Yes, it's very frustrating, I think we're seeing the same issue, I corrected my statement, it affects face/barcode/text detection, and YES, it's inconsistent some devices are fine. (My friend's Huawei phone is fine, on a simulator with a webcam attached is also fine. Pixel phone and Samsung Galaxy tab do not work)

darshankawar commented 4 years ago

Since the devices on which I tried don't have any issues, but per OP, it was working previously on same devices but issue seem to be happening on certain devices like Pixel and Samsung Galaxy tab. Hence keeping this issue open for further analysis. Removed regression label since I wasn't able to replicate in current version.

zean00 commented 4 years ago

Hi, I'm experiencing similar issue with Sony Xperia XZ2 (Android 10) Compact and Oppo A7 (Android 8), both cannot detect face whilst the text detection is still working (but inconsistent). Then I tried to change the camera resolution into ResolutionPreset.high and it's working (face is detected including landmark and contour) . This is no issue on Xperia XZ2 since it has SD845, but it feels lagging when it run on Oppo A7. Before migrating into firebase_core > 0.5.0 this is not an issue, but somehow I couldn't go back into 0.4.0

tindalida commented 4 years ago

Hi, I'm experiencing similar issue with Sony Xperia XZ2 (Android 10) Compact and Oppo A7 (Android 8), both cannot detect face whilst the text detection is still working (but inconsistent). Then I tried to change the camera resolution into ResolutionPreset.high and it's working (face is detected including landmark and contour) . This is no issue on Xperia XZ2 since it has SD845, but it feels lagging when it run on Oppo A7. Before migrating into firebase_core > 0.5.0 this is not an issue, but somehow I couldn't go back into 0.4.0

Thank you so much for the workaround @zean00 , I tested with ResolutionPreset.high and the face recognition worked on my Samsung S20 FE and also on some device I tested such as One Plus 6. Though, not all devices can handle this performance.. :(

What could possibly be causing this, could it be something with the camera plug in?

pixnbit commented 4 years ago

No way!!! Yes, I can confirm after changing from ResolutionPreset.medium to ResolutionPreset.high fixed my issue. It's might be to do with the aspect ratio and buffer sizes. "medium" is not 16:9 but 3:2 on android

Please fix this Flutter fire team.

srombs commented 3 years ago

I am using a pixel 3xl and was experiencing lack of detection for all modes for CameraPreviewScanner. Upping the resolution to ResolutionPreset.high as stated above fixed my problem.

Salakar commented 3 years ago

Hey, :wave:, the firebase_ml_vision package is now discontinued since its APIs have been deprecated and removed from the Android & iOS Firebase SDKs.

I'd recommend switching to the alternatives now;

for on-device vision APIs; switch to Google's standalone ML Kit library via google_ml_kit
for cloud vision APIs; the recommended approach now is to use Firebase Authentication and Functions, which gives you a managed, serverless gateway to the Google Cloud Vision APIs. For an example Functions project see the vision-annotate-images sample project. You'd deploy this project for example and then call the cloud function from your app with your image data to get the annotation result.

Apologies for any inconvenience here and best of luck switching over the the new APIs. firebase_ml_custom is not affected by this deprecation.

Thanks

firebase / flutterfire