Convert to FirebaseVisionImage

[gmparker2000]

Hoping someone might know how to convert to a FirebaseVisionImage. I have the following code that is trying to load an image from assets, crop it, then extract text from it using a firebase TextRecognizer. If I load the FirebaseVisionImage from a file, then everything works. If I crop the image, write it to a file, and then load it into a FirebaseVisionImage, it works. However, trying to load the FirebaseVisionImage directly from the cropped image bytes fails. Not exactly sure what I get back when calling getBytes from the image? Is it RGBA, or something else? I did try encoding as JPG but no luck.

rootBundle.load('assets/test.jpg').then((bytes) {
  imglib.Image image = imglib.decodeImage(bytes.buffer.asUint8List());
  imglib.Image cropped = imglib.copyCrop(image, 84, 1400, 2000, 300);

  var planeData = FirebaseVisionImagePlaneMetadata(
    bytesPerRow: image.width.toInt() * 4,
    height: image.height.toInt(),
    width: image.width.toInt(),
  );

  var meta = FirebaseVisionImageMetadata(
    rawFormat: 'RGBA',
    size: Size(image.width.toDouble(), image.height.toDouble()),
    rotation: ImageRotation.rotation0,
    planeData: [planeData, planeData, planeData, planeData],
  );

  var visionImage = FirebaseVisionImage.fromBytes(
      cropped.getBytes(), meta);

  final TextRecognizer textRecognizer = FirebaseVision.instance.textRecognizer();
  textRecognizer.processImage(visionImage).then((VisionText visionText) {
    for (TextBlock block in visionText.blocks) {
      final String text = block.text;
      final List<RecognizedLanguage> languages = block.recognizedLanguages;
      print(text);
    }
  });

[brendan-duncan]

I don't know the Firebase API at all, or what it's expecting for image data. Image.getBytes is in the format R8G8B8A8. I suspect FirebaseVisionImage only supports very specific formats. According to the documentation for FirebaseVisionImage.fromBytes in https://github.com/FirebaseExtended/flutterfire/blob/master/packages/firebase_ml_vision/lib/src/firebase_vision.dart:

  /// On Android, expects `android.graphics.ImageFormat.NV21` format. Note:
  /// Concatenating the planes of `android.graphics.ImageFormat.YUV_420_888`
  /// into a single plane, converts it to `android.graphics.ImageFormat.NV21`.
  ///
  /// On iOS, expects `kCVPixelFormatType_32BGRA` format. However, this should
  /// work with most formats from `kCVPixelFormatType_*`.

Seems like the expected format is platform dependent. Seems like iOS is more flexible and you could probably get it to work, possibly with a remapColors to shuffle the image channels around to an order it likes. But Android seems to specifically wants the NV21 format, which there is nothing in the this library that can convert to.

You could encode the image to a jpg or png, write it to a tmp file, and then use FirebaseVisionImage.fromFile.

[gmparker2000]

I've been looking for something to convert to NV21 from RGB but there doesn't seem to be anything out there. Thanks for the quick response.

[brendan-duncan]

Here is some code to convert NV21 to RGB. Should be possible to invert that. https://www.codepool.biz/nv21-bmp-java.html

[ozexpert]

I'm also looking for to convert Image to nv21 format. not sure of how those images work, if someone can look into this, it would be very helpful.

[youssefali424]

check this worked fine with google ml kit

Uint8List convertImageToYUV420SP(img.Image image) {
  int width = image.width;
  int height = image.height;
  final int frameSize = width * height;
  var f = height * width + 2 * (height / 2.0).ceil() * (width / 2.0).ceil();
  Uint8List yuv420sp = Uint8List(f);
  var yp = 0;
  for (var p in image) {
    final j = p.y;
    final i = p.x;
    int r = p.r.toInt();
    int g = p.g.toInt();
    int b = p.b.toInt();
    yuv420sp[yp] = ((66 * r + 129 * g + 25 * b + 128) >> 8) + 16;
    if (j % 2 == 0 && i % 2 == 0) {
      int uvp = frameSize + (j >> 1) * width + i;
      yuv420sp[uvp] = ((-38 * r - 74 * g + 112 * b + 128) >> 8) + 128;
      yuv420sp[uvp + 1] = ((112 * r - 94 * g - 18 * b + 128) >> 8) + 128;
    }
    yp++;
  }
  return yuv420sp;
}

[ebsangam]

check this worked fine with google ml kit

Uint8List convertImageToYUV420SP(img.Image image) {
  int width = image.width;
  int height = image.height;
  final int frameSize = width * height;
  var f = height * width + 2 * (height / 2.0).ceil() * (width / 2.0).ceil();
  Uint8List yuv420sp = Uint8List(f);
  var yp = 0;
  for (var p in image) {
    final j = p.y;
    final i = p.x;
    int r = p.r.toInt();
    int g = p.g.toInt();
    int b = p.b.toInt();
    yuv420sp[yp] = ((66 * r + 129 * g + 25 * b + 128) >> 8) + 16;
    if (j % 2 == 0 && i % 2 == 0) {
      int uvp = frameSize + (j >> 1) * width + i;
      yuv420sp[uvp] = ((-38 * r - 74 * g + 112 * b + 128) >> 8) + 128;
      yuv420sp[uvp + 1] = ((112 * r - 94 * g - 18 * b + 128) >> 8) + 128;
    }
    yp++;
  }
  return yuv420sp;
}

YUV420SP is referred as nv21 so make sure you pick nv21 and bytesPerRow set to image width.