search

flutter / flutter

Flutter makes it easy and fast to build beautiful apps for mobile and beyond
https://flutter.dev
BSD 3-Clause "New" or "Revised" License
161.86k stars 26.57k forks source link

[camera_android_camerax] Cannot set imageFormatGroup to ImageFormatGroup.nv21 #145961

Open CaoGiaHieu-dev opened 1 month ago

CaoGiaHieu-dev commented 1 month ago

Steps to reproduce

init camera android by using code below 

 _controller = CameraController(
      camera,
      ResolutionPreset.high,
      enableAudio: false,
      imageFormatGroup: Platform.isAndroid
          ? ImageFormatGroup.nv21
          : ImageFormatGroup.bgra8888,
    );
   _controller?.startImageStream(_processCameraImage);

Expected results

_processCameraImage(InputImage image){
if (_controller == null) return null;
    InputImageRotation? rotation;
    if (Platform.isIOS) {
      rotation = InputImageRotationValue.fromRawValue(sensorOrientation);
    } else if (Platform.isAndroid) {
      var rotationCompensation =
          _orientations[_controller!.value.deviceOrientation];
      if (rotationCompensation == null) return null;
      if (camera.lensDirection == CameraLensDirection.front) {
        rotationCompensation = (sensorOrientation + rotationCompensation) % 360;
      } else {
        rotationCompensation =
            (sensorOrientation - rotationCompensation + 360) % 360;
      }
      rotation = InputImageRotationValue.fromRawValue(rotationCompensation);
    }
    if (rotation == null) return null;
    final format = InputImageFormatValue.fromRawValue(image.format.raw);
    print(format); => return format should return as ImageFormatGroup.nv21InputImageFormat.nv21
}

Actual results

format return InputImageFormat.yuv_420_888

Code sample

Code sample Using code [example of google ml kit](https://github.com/flutter-ml/google_ml_kit_flutter/tree/develop/packages/example) and add camera_android_camerax ### Screenshots or Video
Screenshots / Video demonstration [Upload media here]
### Logs
Logs ```console [Paste your logs here] ```
### Flutter Doctor output
Doctor output ```console [Paste your output here] ```
darshankawar commented 4 weeks ago

@CaoGiaHieu-dev Can you provide flutter doctor -v ? Also, please provide a runnable reproducible code sample that we can directly copy paste and run to verify the reported behavior.

CaoGiaHieu-dev commented 4 weeks ago

flutter doctor -v 

caogiahieu@ML-caogiahieu ekyc_ekyb % flutter doctor -v
[✓] Flutter (Channel stable, 3.19.5, on macOS 14.4 23E214 darwin-x64, locale vi)
    • Flutter version 3.19.5 on channel stable at /Users/caogiahieu/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 300451adae (4 days ago), 2024-03-27 21:54:07 -0500
    • Engine revision e76c956498
    • Dart version 3.3.3
    • DevTools version 2.31.1

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/caogiahieu/Library/Android/sdk
    • Platform android-34, build-tools 31.0.0
    • ANDROID_HOME = /Users/caogiahieu/Library/Android/sdk
    • ANDROID_SDK_ROOT = /Users/caogiahieu/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15E204a
    • CocoaPods version 1.15.2

[✓] Android Studio (version 2023.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)

[✓] VS Code (version 1.87.2)
    • VS Code at /Users/caogiahieu/Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.84.0

[!] Connected device
    ! No devices available

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.

Reproducible code

```console CameraController? _cameraController; List cameras=[]; CameraDescription get currentCamera => cameras.firstWhere( (element) => element.lensDirection == CameraLensDirection.front, ); @override void initState() { WidgetsBinding.instance.endOfFrame.whenComplete(() async { final res = await availableCameras(); cameras.addAll( [ res.firstWhere( (element) => element.lensDirection == CameraLensDirection.front, ), res.firstWhere( (element) => element.lensDirection == CameraLensDirection.back, ), ], ); _cameraController = CameraController( currentCamera, ResolutionPreset.veryHigh, enableAudio: false, imageFormatGroup: Platform.isAndroid ? ImageFormatGroup.nv21 : ImageFormatGroup.bgra8888, ); await _cameraController?.initialize(); _cameraController?.startImageStream( _inputImageFromCameraImage , ); }); super.initState(); } final _orientations = { DeviceOrientation.portraitUp: 0, DeviceOrientation.landscapeLeft: 90, DeviceOrientation.portraitDown: 180, DeviceOrientation.landscapeRight: 270, }; void _inputImageFromCameraImage(CameraImage image) { if (_cameraController == null) return; final sensorOrientation = currentCamera.sensorOrientation; InputImageRotation? rotation; if (Platform.isIOS) { rotation = InputImageRotationValue.fromRawValue(sensorOrientation); } else if (Platform.isAndroid) { var rotationCompensation = _orientations[_cameraController!.value.deviceOrientation]; if (rotationCompensation == null) return; if (currentCamera.lensDirection == CameraLensDirection.front) { rotationCompensation = (sensorOrientation + rotationCompensation) % 360; } else { rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360; } rotation = InputImageRotationValue.fromRawValue(rotationCompensation); } if (rotation == null) return; final format = InputImageFormatValue.fromRawValue(image.format.raw); print(format); // in Android it should be InputImageFormat.nv21 instead InputImageFormat.yuv_420_888 base on config } } ```

It work normal with package camera_android . but not in camera_android_camerax

darshankawar commented 3 weeks ago

Thanks for the update. I was able to replicate this using above code sample running on S10.

stable, master flutter doctor -v ``` [!] Flutter (Channel stable, 3.19.5, on macOS 12.2.1 21D62 darwin-x64, locale en-GB) • Flutter version 3.19.5 on channel stable at /Users/dhs/documents/fluttersdk/flutter ! Warning: `flutter` on your path resolves to /Users/dhs/Documents/Fluttersdk/flutter/bin/flutter, which is not inside your current Flutter SDK checkout at /Users/dhs/documents/fluttersdk/flutter. Consider adding /Users/dhs/documents/fluttersdk/flutter/bin to the front of your path. ! Warning: `dart` on your path resolves to /Users/dhs/Documents/Fluttersdk/flutter/bin/dart, which is not inside your current Flutter SDK checkout at /Users/dhs/documents/fluttersdk/flutter. Consider adding /Users/dhs/documents/fluttersdk/flutter/bin to the front of your path. • Upstream repository https://github.com/flutter/flutter.git • Framework revision 300451adae (4 days ago), 2024-03-27 21:54:07 -0500 • Engine revision e76c956498 • Dart version 3.3.3 • DevTools version 2.31.1 • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades. [!] Xcode - develop for iOS and macOS (Xcode 12.3) • Xcode at /Applications/Xcode.app/Contents/Developer ! Flutter recommends a minimum Xcode version of 13. Download the latest version or update via the Mac App Store. • CocoaPods version 1.11.2 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] VS Code (version 1.62.0) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.21.0 [✓] Connected device (5 available) • SM G975F (mobile) • RZ8M802WY0X • android-arm64 • Android 11 (API 30) • Darshan's iphone (mobile) • 21150b119064aecc249dfcfe05e259197461ce23 • ios • iOS 14.4.1 18D61 • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729 • ios • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator) • macOS (desktop) • macos • darwin-x64 • Mac OS X 10.15.4 19E2269 darwin-x64 • Chrome (web) • chrome • web-javascript • Google Chrome 98.0.4758.80 [✓] HTTP Host Availability • All required HTTP hosts are available ! Doctor found issues in 1 category. [!] Flutter (Channel master, 3.22.0-1.0.pre.37, on macOS 12.2.1 21D62 darwin-x64, locale en-GB) • Flutter version 3.22.0-1.0.pre.37 on channel master at /Users/dhs/documents/fluttersdk/flutter ! Warning: `flutter` on your path resolves to /Users/dhs/Documents/Fluttersdk/flutter/bin/flutter, which is not inside your current Flutter SDK checkout at /Users/dhs/documents/fluttersdk/flutter. Consider adding /Users/dhs/documents/fluttersdk/flutter/bin to the front of your path. ! Warning: `dart` on your path resolves to /Users/dhs/Documents/Fluttersdk/flutter/bin/dart, which is not inside your current Flutter SDK checkout at /Users/dhs/documents/fluttersdk/flutter. Consider adding /Users/dhs/documents/fluttersdk/flutter/bin to the front of your path. • Upstream repository https://github.com/flutter/flutter.git • Framework revision fd8561a917 (8 hours ago), 2024-03-31 17:12:24 -0400 • Engine revision 4f6b832c8e • Dart version 3.5.0 (build 3.5.0-5.0.dev) • DevTools version 2.34.1 • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades. [!] Android toolchain - develop for Android devices (Android SDK version 30.0.3) • Android SDK at /Users/dhs/Library/Android/sdk ✗ cmdline-tools component is missing Run `path/to/sdkmanager --install "cmdline-tools;latest"` See https://developer.android.com/studio/command-line for more details. ✗ Android license status unknown. Run `flutter doctor --android-licenses` to accept the SDK licenses. See https://flutter.dev/docs/get-started/install/macos#android-setup for more details. [✓] Xcode - develop for iOS and macOS (Xcode 13.2.1) • Xcode at /Applications/Xcode.app/Contents/Developer • Build 13C100 • CocoaPods version 1.11.2 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] IntelliJ IDEA Ultimate Edition (version 2021.3.2) • IntelliJ at /Applications/IntelliJ IDEA.app • Flutter plugin version 65.1.4 • Dart plugin version 213.7228 [✓] VS Code (version 1.62.0) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.29.0 [✓] Connected device (3 available) • Darshan's iphone (mobile) • 21150b119064aecc249dfcfe05e259197461ce23 • ios • iOS 15.3.1 19D52 • macOS (desktop) • macos • darwin-x64 • macOS 12.2.1 21D62 darwin-x64 • Chrome (web) • chrome • web-javascript • Google Chrome 109.0.5414.119 [✓] Network resources • All expected network resources are available. ! Doctor found issues in 1 category. [!] Xcode - develop for iOS and macOS (Xcode 12.3) • Xcode at /Applications/Xcode.app/Contents/Developer ! Flutter recommends a minimum Xcode version of 13. Download the latest version or update via the Mac App Store. • CocoaPods version 1.11.2 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] VS Code (version 1.62.0) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.21.0 [✓] Connected device (5 available) • SM G975F (mobile) • RZ8M802WY0X • android-arm64 • Android 11 (API 30) • Darshan's iphone (mobile) • 21150b119064aecc249dfcfe05e259197461ce23 • ios • iOS 14.4.1 18D61 • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729 • ios • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator) • macOS (desktop) • macos • darwin-x64 • Mac OS X 10.15.4 19E2269 darwin-x64 • Chrome (web) • chrome • web-javascript • Google Chrome 98.0.4758.80 [✓] HTTP Host Availability • All required HTTP hosts are available ! Doctor found issues in 1 category. ```
Rethen commented 3 weeks ago

Because in the code of camera_android_camerax, there is no image conversion code at all

Rethen commented 3 weeks ago

ndroid_camerax, there is no image conversion c

感谢更新。我能够使用在 S10 上运行的上述代码示例来复制此内容。

稳定，master flutter doctor -v

Please add the code related to format conversion as soon as possible. We need to use nv21 images for object detection.

Rethen commented 3 weeks ago

Thanks for the update. I was able to replicate this using above code sample running on S10.

stable, master flutter doctor -v

Will you fix this problem in the near future?

Rethen commented 2 weeks ago

I very much hope that the flutter team can handle this issue at the highest level, I think this is a missing feature, which directly affects everyone who wants to use this library for image analysis such as tensorflow or Yolo.

reidbaker commented 1 week ago

See https://github.com/flutter/flutter/wiki/Issue-hygiene#when-will-my-bug-be-fixed

We have no specific plan to work on this, so there is no ETA. Anyone interested is welcome to submit a PR.