Unable to Detect Poses on Recorded Video using InputImage.fromBytes()

[HasithMbiz]

Poses are drawing on the live camera feed without an issue. But I want to draw poses on a recorded video. I can take snapshots of the video frames as bytes (Uint8List). Then what should I pass as InputImageMetadata. I tried pass default values similar to what is applied from the CameraImage to the InputImage during live camera in my device. Like below,

final inputImage = InputImage.fromBytes(
      bytes: imageBytes,
      metadata: InputImageMetadata(
        size: const Size(1280, 720),
        rotation: InputImageRotation.rotation270deg,
        format: InputImageFormat.nv21,
        bytesPerRow: 1280,
      ),
    );

But then I encounter the following error when process the image.

final poses = await _poseDetector.processImage(inputImage);

[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: PlatformException(error, Image dimension, ByteBuffer size and format don't match. Please check if the ByteBuffer is in the decalred format., null, java.lang.IllegalArgumentException: Image dimension, ByteBuffer size and format don't match. Please check if the ByteBuffer is in the decalred format. at com.google.android.gms.common.internal.Preconditions.checkArgument(com.google.android.gms:play-services-basement@@18.3.0:2) at com.google.mlkit.vision.common.InputImage.(com.google.mlkit:vision-common@@17.3.0:10) at com.google.mlkit.vision.common.InputImage.fromByteArray(com.google.mlkit:vision-common@@17.3.0:2) at com.google_mlkit_commons.InputImageConverter.getInputImageFromData(InputImageConverter.java:36) at com.google_mlkit_pose_detection.PoseDetector.handleDetection(PoseDetector.java:52) at com.google_mlkit_pose_detection.PoseDetector.onMethodCall(PoseDetector.java:38) at io.flutter.plugin.common.MethodChannel$IncomingMethodCallHandler.onMessage(MethodChannel.java:267) at io.flutter.embedding.engine.dart.DartMessenger.invokeHandler(DartMessenger.java:292) at io.flutter.embedding.engine.dart.DartMessenger.lambda$dispatchMessageToQueue$0$io-flutter-embedding-engine-dart-DartMessenger(DartMessenger.java:319) at io.flutter.embedding.engine.dart.DartMessenger$$ExternalSyntheticLambda0.run(Unknown Source:12) at android.os.Handler.handleCallback(Handler.java:883) at android.os.Handler.dispatchMessage(Handler.java:100) at android.os.Looper.loop(Looper.java:214) at android.app.ActivityThread.main(ActivityThread.java:7355) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:914) )

How can I use Uint8List to detect the poses. I did get the poses by using the InputImage.fromFile intead of InputImage.fromBytes. but saving Uint8List into a file take more time and not efficient. Can anyone suggest a solution for this.

[famasf1]

Having the same issue. Apparently this is because the camera for some reason ignore your CameraController.imageFormatGroup setting when received images. And forced it back to YUV420 regardless of what camera image type you picked.

I'm not sure if this is from Flutter (i just upgrade to 3.22), The package itself or the Camera package (they updated it a few days ago.)

Edit : The workaround for now is converting your CameraImage from yuv420 to NV21.

Uint8List _yuv420ToNV21(CameraImage image) {
        var nv21 = Uint8List(image.planes[0].bytes.length +
            image.planes[1].bytes.length +
            image.planes[2].bytes.length);

        var yBuffer = image.planes[0].bytes;
        var uBuffer = image.planes[1].bytes;
        var vBuffer = image.planes[2].bytes;

        nv21.setRange(0, yBuffer.length, yBuffer);

        int i = 0;
        while (i < uBuffer.length) {
          nv21[yBuffer.length + i] = vBuffer[i];
          nv21[yBuffer.length + i + 1] = uBuffer[i];
          i += 2;
        }

        return nv21;
      }

Put it before you return InputImage.FromBytes like this

Uint8List newImg = _yuv420ToNV21(image);
      final plane = image.planes.first;
      final format = InputImageFormatValue.fromRawValue(image.format.raw);
      InputImageRotation? rotation =
          _getInputImgRotation(camera, cameraController);
      if (!_shouldReturnNull(image, rotation, format)) return null;

      return InputImage.fromBytes(
        bytes: newImg,
        metadata: InputImageMetadata(
          size: Size(image.width.toDouble(), image.height.toDouble()),
          rotation: rotation!,
          format: format!,
          bytesPerRow: plane.bytesPerRow,
        ),
      );

[michaelnew]

Relevant issue on the Flutter repo: https://github.com/flutter/flutter/issues/145961

As of now, manually converting to NV21 or using the older camera plugin version 10.6 are the only options.

[ShuheiSuzuki-07]

Anyone. Do you know why it sometimes works with YUV420? Also, are there any devices that will not work by specifying NV21?

[FantaMagier]

@famasf1 What is you _shouldReturnNull Function? Thanks for the Code!

[famasf1]

@famasf1 What is you _shouldReturnNull Function? Thanks for the Code!

It's just a private function for my camera class that check whether the camera should return any value upon detection. If all values inside conditions block return true then execute InputImage.fromBytes. Otherwise return null.

It wasn't necessary, Most of the code is from this lib's example. I just abstract it because i don't like seeing a bunch of if scattering around (there are a lot of if check fail then return null like this in the example code and it bothers me lol)

  bool _shouldReturnNull(
    CameraImage image,
    InputImageRotation? rotation,
    InputImageFormat? format,
  ) {
    List<bool> conditions = [
      rotation != null,
      format != null,
      Platform.isIOS ? image.planes.length == 1 : image.planes.length != 1,
    ];

    return conditions.every((element) => element);
  }

Relevant issue on the Flutter repo: flutter/flutter#145961

As of now, manually converting to NV21 or using the older camera plugin version 10.6 are the only options.

It's crazy that this issues has been up for like 4 months now and there are still no fix in sight.

[FantaMagier]

@fbernaly Thanks for your reply! I think the process to change the ml kit package to the new image format isn't easy even though the android sdk supports YUV420 out of the box. To communicate the native part we have to format the YUV420 to json I guess and then we can send the image to the android sdk way. The flutter team isn't interested in supporting the nv21 format on the officelle camera package unfortunately.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[famasf1]

Having the same issue. Apparently this is because the camera for some reason ignore your CameraController.imageFormatGroup setting when received images. And forced it back to YUV420 regardless of what camera image type you picked.

I'm not sure if this is from Flutter (i just upgrade to 3.22), The package itself or the Camera package (they updated it a few days ago.)

Edit : The workaround for now is converting your CameraImage from yuv420 to NV21.

Uint8List _yuv420ToNV21(CameraImage image) {
        var nv21 = Uint8List(image.planes[0].bytes.length +
            image.planes[1].bytes.length +
            image.planes[2].bytes.length);

        var yBuffer = image.planes[0].bytes;
        var uBuffer = image.planes[1].bytes;
        var vBuffer = image.planes[2].bytes;

        nv21.setRange(0, yBuffer.length, yBuffer);

        int i = 0;
        while (i < uBuffer.length) {
          nv21[yBuffer.length + i] = vBuffer[i];
          nv21[yBuffer.length + i + 1] = uBuffer[i];
          i += 2;
        }

        return nv21;
      }

Put it before you return InputImage.FromBytes like this

Uint8List newImg = _yuv420ToNV21(image);
      final plane = image.planes.first;
      final format = InputImageFormatValue.fromRawValue(image.format.raw);
      InputImageRotation? rotation =
          _getInputImgRotation(camera, cameraController);
      if (!_shouldReturnNull(image, rotation, format)) return null;

      return InputImage.fromBytes(
        bytes: newImg,
        metadata: InputImageMetadata(
          size: Size(image.width.toDouble(), image.height.toDouble()),
          rotation: rotation!,
          format: format!,
          bytesPerRow: plane.bytesPerRow,
        ),
      );

It's worth noting that, As of 3 days ago with the new update from this plugin to version 0.11.1, My code above will not work and will result in error being thrown now.

PlatformException(InputImageConverterError, java.lang.IllegalArgumentException, null, null)

I believed this is the related PR. given that the same error is being thrown for not satisfied correct image format. But i'm not sure either. since according to changelog. It simply 'Update Dependencies.' And didn't actually implementing this PR yet. Plus removing my converted code didn't fix anything. So it might be just CameraX being funny and that i should've picked the downgrading camera plugin method instead.

The only fix i found is downgrade google_ml_kit_flutter. I'm using google_mlkit_face_detection and google_mlkit_object_detection So downgrading to

google_mlkit_face_detection: 0.11.0
google_mlkit_object_detection: 0.13.0

solved my issue. If you didn't downgrade to camera: 0.10.6 yet and picking NV21 conversion method. Then please avoid updating google_ml_kit_flutter until further notice.

[FantaMagier]

@fbernaly Can you take a look on that? Thank You!

[michaelnew]

Yeah same issue; I'm using google_mlkit_text_recognition 13.1, and trying to manually convert from yuv_420_888 to nv21 is causing a InputImageConverterError.

Downgrading to camera 10.6 does still work, so there must be something different between the manually converted nv21 data and the nv21 data returned from the camera package.

The PR linked above is not related I don't think (it's been merged for a while). Even though you can pass yuv_420_888 as an image format to mlkit, we still have to convert from CameraImage to InputImage, and the fromBytes constructor only works with a single byte array, so there's no way to construct the InputImage when we have a multi-plane format like yuv_420.

It seems like the best fix would be to allow for converting a yuv_420_888 CameraImage to an InputImage of the same format, but the hacky fix is probably to figure out why the manually converted nv21 isn't matching with what the old camera package was returning.

[famasf1]

Downgrading to camera 10.6 does still work

It's important to note that this is because Flutter Team out-of-the-blue decided to implemented CameraX into the library and replace the old implementation.

I haven't downgrade Camera and stick with manually converting yuv_420_888 to NV21. and upgrading this lib will caused the error thrown now. Which didn't happened before on previous version. Downgrading Camera is probably the best solution now.

[fbernaly]

when I was working on the latest changes, updating to the latest camera plugin breaks the example app, that is why the example app is still using 0.10.6 https://pub.dev/packages/camera/versions/0.10.6, you can check it here: https://github.com/flutter-ml/google_ml_kit_flutter/blob/develop/packages/example/pubspec.lock#L20C3-L27

I recommend you do not update to camera 0.11.0 or later.

[FantaMagier]

Google is planning to support nv21 for CameraX: https://github.com/flutter/flutter/issues/145961 https://issuetracker.google.com/issues/359664078 But I don't know when this is planned.