BufferOverflowException when using FirebaseModelDataType.FLOAT32 in custom model

[adamkoch]

I see the following crash when using FirebaseModelDataType.FLOAT32 as the input type for my custom model:

2018-10-18 00:15:30.178 25593-25593/com.devrel.sample.fluttermlkit E/MethodChannel#plugins.flutter.io/mlkit: Failed to handle method call
    java.nio.BufferOverflowException
        at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:291)
        at java.nio.ByteBuffer.put(ByteBuffer.java:732)
        at com.azihsoyn.flutter.mlkit.MlkitPlugin.onMethodCall(MlkitPlugin.java:290)
        at io.flutter.plugin.common.MethodChannel$IncomingMethodCallHandler.onMessage(MethodChannel.java:191)
        at io.flutter.view.FlutterNativeView.handlePlatformMessage(FlutterNativeView.java:163)
        at android.os.MessageQueue.nativePollOnce(Native Method)
        at android.os.MessageQueue.next(MessageQueue.java:326)
        at android.os.Looper.loop(Looper.java:160)
        at android.app.ActivityThread.main(ActivityThread.java:6669)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)

I think it might be because not enough memory is allocated in the ByteBuffer here: https://github.com/azihsoyn/flutter_mlkit/blob/8533fa39c662645feff371e1836bc2981a1f8a93/android/src/main/java/com/azihsoyn/flutter/mlkit/MlkitPlugin.java#L287

It looks like only the other input parameters are used when the type should be considered as well (if float then *4 the space needed).

[azihsoyn]

Hi, @adamkoch! Thank you for reporting!

I'll fix it next release. (maybe this week) Or PR is welcome😁

[gildaswise]

@adamkoch Hey, could you show how you're handling image to float conversion?

[adamkoch]

This is what I'm using right now, although I haven't been able to get the end-to-end flow working so I'm not sure if its working correctly:

Uint8List imageToByteListFloat(img.Image image) {
    var convertedBytes = Uint8List(1 * imageSize * imageSize * 3 * 4);
    ByteData buffer = ByteData.view(convertedBytes.buffer);
    try {
      int index = 0;
      for (var i = 0; i < imageSize; i++) {
        for (var j = 0; j < imageSize; j++) {
          var pixel = image.getPixel(i, j);
          buffer.setFloat32(index += 4, ((pixel >> 16) & 0xFF) / 255);
          buffer.setFloat32(index += 4, ((pixel >> 8) & 0xFF) / 255);
          buffer.setFloat32(index += 4, ((pixel) & 0xFF) / 255);
        }
      }
    } catch (e) {
      print('imageToByteList error: $e');
    }
    return convertedBytes;
  }

I actually found another issue with using FLOAT32 here: https://github.com/azihsoyn/flutter_mlkit/blob/master/android/src/main/java/com/azihsoyn/flutter/mlkit/MlkitPlugin.java#L343

The StandardMessage Codec does not seem to support passing a float[] back from Android to Flutter (only double[]) so it was not passing anything back for me.

[gildaswise]

@adamkoch Sadly I'm getting RangeError when using an image of size 224x224.

[adamkoch]

Yeh I don't think the root cause is fixed yet so it won't work. It looks like @azihsoyn is working on it though (thank you!).

[gildaswise]

@adamkoch I mean, it couldn't even convert to a float array; it wasn't an error of MLKit. I haven't found any other implementations of this conversion in Dart.

[adamkoch]

@gildaswise can you paste your code? Or maybe to stackoverflow and link here? I'm no Dart expert so maybe best to ask the experts. One thing though, is if you are converting to float array for flutter mlkit library I think that might be unnecessary. As per my code above, it should work fine to create a Uint8List instead but 4x the size (to fit the floats) and then just write the floats to the the raw byte array (buffer).

[gildaswise]

@adamkoch

  static Uint8List imageToByteListFloat(img.Image image, int size) {
    Logger.log(TAG, message: "Converting to Uint8List an image of size: $size");
    final convertedBytes = Uint8List((1 * size * size * 3) * 4);
    final buffer = ByteData.view(convertedBytes.buffer);
    int index = 0;
    try {
      for (var i = 0; i < size; i++) {
        for (var j = 0; j < size; j++) {
          var pixel = image.getPixel(i, j);
            buffer.setFloat32(index += 4, ((pixel >> 16) & 0xFF) / 255);
            buffer.setFloat32(index += 4, ((pixel >> 8) & 0xFF) / 255);
            buffer.setFloat32(index += 4, ((pixel) & 0xFF) / 255);
        }
      }
      return convertedBytes;
    } catch (error) {
      Logger.log(TAG,
          message: "Couldn't convert to Float32 Uint8List, error: $error");
      return null;
    }
  }

I'm just using your method in this other:

static Uint8List prepareAnalysis(ModelPayload payload) {
    img.Image resized = img.decodeJpg(payload.image.readAsBytesSync());
    Logger.log(TAG,
        message:
            "Loaded image's dimensions: ${resized.height}x${resized.width}");
    if (resized.height != payload.desiredSize ||
        resized.width != payload.desiredSize)
      resized =
          img.copyResize(resized, payload.desiredSize, payload.desiredSize);
    Logger.log(TAG,
        message:
            "Resized image's dimensions: ${resized.height}x${resized.width}");
    return (payload.isFloat)
        ? imageToByteListFloat(resized, payload.desiredSize)
        : imageToByteList(resized, payload.desiredSize);
  }

And ModelPayload is this:

class ModelPayload {
  final File image;
  final int desiredSize;
  final bool isFloat;

  ModelPayload(this.image, this.desiredSize, {this.isFloat = false});
}

so I can send this to an Isolate.

Then I get this: flutter: [IMAGE_UTILS] Couldn't convert to Float32 Uint8List, error: RangeError (byteOffset): Invalid value: Not in range 0..602108, inclusive: 602112

I don't even know how that's happening as I also tried to break the loop when reaching the max index. 602112 is the value resulted from (1 * size * size * 3) * 4.

[azihsoyn]

Now I'm working fix to this, but it takes a time. In my estimate, I can release until next weekend.

[adamkoch]

Thanks so much @azihsoyn!

@gildaswise as a float32 takes up 4 bytes, the max index you can write a float to in the uint8list will be 602108 (if length is 602112). i thought my logic was correct but you should double check that the loops do stop (and index does end up) at the right value so it only writes the last float to 602108.

[azihsoyn]

Hi, I released 0.9.0 to fix this.

To use the float32 model, please try below code.(also documented README)

// float model
Uint8List imageToByteList(img.Image image) {
  var _inputSize = 224;
  var convertedBytes = Float32List(1 * _inputSize * _inputSize * 3);
  var buffer = Float32List.view(convertedBytes.buffer);
  int pixelIndex = 0;
  for (var i = 0; i < _inputSize; i++) {
    for (var j = 0; j < _inputSize; j++) {
      var pixel = image.getPixel(i, j);
      buffer[pixelIndex] = ((pixel >> 16) & 0xFF) / 255;
      pixelIndex += 1;
      buffer[pixelIndex] = ((pixel >> 8) & 0xFF) / 255;
      pixelIndex += 1;
      buffer[pixelIndex] = ((pixel) & 0xFF) / 255;
      pixelIndex += 1;
    }
  }
  return convertedBytes.buffer.asUint8List();
}

And flutter cannot convert float32 from android, so return byte array as a workaround. You need to convert to float32 list in the flutter.

Thanks.

[Baileypollard]

Hi, I released 0.9.0 to fix this.

To use the float32 model, please try below code.(also documented README)

// float model
Uint8List imageToByteList(img.Image image) {
  var _inputSize = 224;
  var convertedBytes = Float32List(1 * _inputSize * _inputSize * 3);
  var buffer = Float32List.view(convertedBytes.buffer);
  int pixelIndex = 0;
  for (var i = 0; i < _inputSize; i++) {
    for (var j = 0; j < _inputSize; j++) {
      var pixel = image.getPixel(i, j);
      buffer[pixelIndex] = ((pixel >> 16) & 0xFF) / 255;
      pixelIndex += 1;
      buffer[pixelIndex] = ((pixel >> 8) & 0xFF) / 255;
      pixelIndex += 1;
      buffer[pixelIndex] = ((pixel) & 0xFF) / 255;
      pixelIndex += 1;
    }
  }
  return convertedBytes.buffer.asUint8List();
}

And flutter cannot convert float32 from android, so return byte array as a workaround. You need to convert to float32 list in the flutter.

Thanks.

Have you tested this? I'm attempting to use a float32 model, but it isn't returning the expected results