Process Camera Image Issue when using Face Detection & Pose Detection,etc with camera_android_camerax: any instead of camera_android:any

[mrwebbeast]

Title: Process Camera Image Issue

Describe the bug Process Camera Image Issue when using Face Detection & Pose Detection,etc with camera_android_camerax: any

To Reproduce Steps to reproduce the behavior:

1. Use default Example Code
2. Use camera: ^0.10.6 & Add camera_android_camerax: ^0.6.4+1
3. Run The App and Face Detection & Pose Detection will not work

Expected behavior It should work same as camera_android: ^any so it will process and return processed data according to mlkit plugin Expected InputImageFormat is nv21 but when using camera_android_camerax: any then its returning yuv_420_888 InputImageFormat

Need support for yuv_420_888 InputImageFormat when using camera_android_camerax:any

Platform (please complete the following information):

• OS: Android 14
• Device: Redmi Note 12 Pro Plus
• Flutter/Dart Version 3.19.6 • channel stable | Dart 3.3.4
  • Plugin version 1) google_mlkit_face_detection: ^0.11.0 2) google_mlkit_pose_detection: ^0.12.0

[fbernaly]

feel free to fork the repo and edit this:

https://github.com/flutter-ml/google_ml_kit_flutter/blob/develop/packages/google_mlkit_commons/android/src/main/java/com/google_mlkit_commons/InputImageConverter.java#L35-L41

send you PR back with your contribution

[FantaMagier]

CameraX is now the default Android camera in the Camera Package. It would be great if we could get the update.

[AndreiMisiukevich]

I believe yuv_420_888 format can have more than a single plane. It's unclear how to build an InputImage then. (In previous versions, probably 0.0.6 it worked with multiple planes, but it seems not anymore).

Also, the documentation of this plugin states to use `.nv21 for android.

[fbernaly]

@AndreiMisiukevich : feel free to fork the repo and add multiple planes and add yuv_420_888 for Android. Then send your PR. We always welcome contributions.

[fbernaly]

Building picture-perfect camera experiences in Flutter with CameraX

[FantaMagier]

This topic is also linked to this CameraX issue: https://github.com/flutter/flutter/issues/145961

[FantaMagier]

Isn't it possible to use the ML KIT API with yuv_420_888 InputImageFormat? Maybe it is easier to support this third image format in the Flutter SDK?

https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage.ImageFormat

[fbernaly]

@FantaMagier : yes, it is possible. I do not have an ETA for that change. We need a volunteer to work on that. Contributions are always welcome.

[PudovkinSergey]

Maybe someone need a quick solution, this code helped me to convert yuv420_888 to nv21 manually: (source in java: https://blog.minhazav.dev/how-to-use-renderscript-to-convert-YUV_420_888-yuv-image-to-bitmap/#tonv21image-image-java-approach)

extension Nv21Converter on CameraImage {
  Uint8List getNv21Uint8List() {
    final width = this.width;
    final height = this.height;

    final yPlane = planes[0];
    final uPlane = planes[1];
    final vPlane = planes[2];

    final yBuffer = yPlane.bytes;
    final uBuffer = uPlane.bytes;
    final vBuffer = vPlane.bytes;

    final numPixels = (width * height * 1.5).toInt();
    final nv21 = List<int>.filled(numPixels, 0);

    // Full size Y channel and quarter size U+V channels.
    int idY = 0;
    int idUV = width * height;
    final uvWidth = width ~/ 2;
    final uvHeight = height ~/ 2;
    // Copy Y & UV channel.
    // NV21 format is expected to have YYYYVU packaging.
    // The U/V planes are guaranteed to have the same row stride and pixel stride.
    // getRowStride analogue??
    final uvRowStride = uPlane.bytesPerRow;
    // getPixelStride analogue
    final uvPixelStride = uPlane.bytesPerPixel ?? 0;
    final yRowStride = yPlane.bytesPerRow;
    final yPixelStride = yPlane.bytesPerPixel ?? 0;

    for (int y = 0; y < height; ++y) {
      final uvOffset = y * uvRowStride;
      final yOffset = y * yRowStride;

      for (int x = 0; x < width; ++x) {
        nv21[idY++] = yBuffer[yOffset + x * yPixelStride];

        if (y < uvHeight && x < uvWidth) {
          final bufferIndex = uvOffset + (x * uvPixelStride);
          //V channel
          nv21[idUV++] = vBuffer[bufferIndex];
          //V channel
          nv21[idUV++] = uBuffer[bufferIndex];
        }
      }
    }
    return Uint8List.fromList(nv21);
  }
}

[FantaMagier]

@PudovkinSergey Good work! How does your inputImageFromCameraImage function look like?

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[nebis-software]

same issue here 😢

[hiroppi401]

Maybe someone need a quick solution, this code helped me to convert yuv420_888 to nv21 manually: (source in java: https://blog.minhazav.dev/how-to-use-renderscript-to-convert-YUV_420_888-yuv-image-to-bitmap/#tonv21image-image-java-approach)

extension Nv21Converter on CameraImage {
  Uint8List getNv21Uint8List() {
    final width = this.width;
    final height = this.height;

    final yPlane = planes[0];
    final uPlane = planes[1];
    final vPlane = planes[2];

    final yBuffer = yPlane.bytes;
    final uBuffer = uPlane.bytes;
    final vBuffer = vPlane.bytes;

    final numPixels = (width * height * 1.5).toInt();
    final nv21 = List<int>.filled(numPixels, 0);

    // Full size Y channel and quarter size U+V channels.
    int idY = 0;
    int idUV = width * height;
    final uvWidth = width ~/ 2;
    final uvHeight = height ~/ 2;
    // Copy Y & UV channel.
    // NV21 format is expected to have YYYYVU packaging.
    // The U/V planes are guaranteed to have the same row stride and pixel stride.
    // getRowStride analogue??
    final uvRowStride = uPlane.bytesPerRow;
    // getPixelStride analogue
    final uvPixelStride = uPlane.bytesPerPixel ?? 0;
    final yRowStride = yPlane.bytesPerRow;
    final yPixelStride = yPlane.bytesPerPixel ?? 0;

    for (int y = 0; y < height; ++y) {
      final uvOffset = y * uvRowStride;
      final yOffset = y * yRowStride;

      for (int x = 0; x < width; ++x) {
        nv21[idY++] = yBuffer[yOffset + x * yPixelStride];

        if (y < uvHeight && x < uvWidth) {
          final bufferIndex = uvOffset + (x * uvPixelStride);
          //V channel
          nv21[idUV++] = vBuffer[bufferIndex];
          //V channel
          nv21[idUV++] = uBuffer[bufferIndex];
        }
      }
    }
    return Uint8List.fromList(nv21);
  }
}

how to use it?