TFlite model loading and detecting in Flutter

[NeighborhoodCoding]

Hi, cuz I'm using in flutter

referring by this: https://pub.dev/packages/tflite

I just used below command in flutter

res = await Tflite.loadModel(
          model: "assets/yolov4-416-fp32.tflite",
          labels: "assets/coco.txt",
        );

var recognitions = await Tflite.detectObjectOnImage(
        path: image.path,
        model: "YOLO",
        threshold: 0.3,
        imageMean: 0.0,
        imageStd: 255.0,
        numResultsPerClass: 1);

But there was an error.

---

Caused by: java.lang.IllegalArgumentException: Cannot copy from a TensorFlowLite tensor (Identity) with shape [1, 2535, 4] to a Java object with shape [1, 13, 13, 425]

---

something more i type correctly in Tflite.detectObjectOnFrame?

[TexMexMax]

Hi, cuz I'm using in flutter

referring by this: https://pub.dev/packages/tflite

I just used below command in flutter

res = await Tflite.loadModel(
          model: "assets/yolov4-416-fp32.tflite",
          labels: "assets/coco.txt",
        );

var recognitions = await Tflite.detectObjectOnImage(
        path: image.path,
        model: "YOLO",
        threshold: 0.3,
        imageMean: 0.0,
        imageStd: 255.0,
        numResultsPerClass: 1);

But there was an error.

Caused by: java.lang.IllegalArgumentException: Cannot copy from a TensorFlowLite tensor (Identity) with shape [1, 2535, 4] to a Java object with shape [1, 13, 13, 425]

something more i type correctly in Tflite.detectObjectOnFrame?

Did you get it to work on flutter? Looking to get yolov4 model to work on flutter too

[NeighborhoodCoding]

@TexMexMax

I actually found that TFlite plugin for flutter dose not support yolov4 for now(but the author said that he'll support someday) and I'm now trying to import my own neural network in the android app using java, but it is not finished yet.(Java is really hard for me...) https://github.com/shaqian/flutter_tflite/issues/42 https://github.com/shaqian/flutter_tflite/issues/44

Maybe if you want to import custom model of yolov4 in android now, you may be better to search another github code, not flutter especially because I think yolov4 is not yet imported in flutter.. maybe java or kotlin have successed?

[TexMexMax]

Actually, if you use the tflite plugin, it doesn't. However, there is another flutter plugin called tflite_flutter. It uses the C bindings for the TensorFlow lite C API. This guy basically created an entire API for Dart. It is actually better than the Tensorflow lite Java API since the C API has less latency. The plugin is located in the link below:

https://pub.dev/packages/tflite_flutter

This guy even wrote a support library for this plugin called tflite_flutter_helper. The link is below: https://pub.dev/packages/tflite_flutter_helper

I build an app that uses a yolov4 model by merely looking at the functions getDetectionsForFull(...) and getDetectionsForTiny(...) in this repository as well as looking at this repository made by the guy who made the above plugins. https://github.com/am15h/object_detection_flutter/blob/master/lib/tflite/classifier.dart

Just tweaked the that class a bit and it worked.

This is the resulting code: Edited: Posted code was wrong. Look below for much better version of the code.

[NeighborhoodCoding]

Thank you very much. I just started working at the company and I was mainly used Python, when I want to use neural network to the app, it was very difficult because it required knowledge of Android API and Java grammar. That's why I see flutter in first, but I was frustrated that flutter also needed some low code to connect the neural network, and I tried a little bit in JAVA... I finished model converting, loading, inputting but I didn't know how to display the outputs of neural network in display, because I don't know well in java data formats... (I was actually trying Korean OCR... Also recently I have worked another task in company so I paused above things for some weeks)

Thank you for the answer above and I will study the above code using C API. Thank you! And seems https://pub.dev/packages/tflite_flutter_helper also will be really great for others. Thank you! Thanks for your help, yolo v4 will work greatly in flutter!!

[Liisjak]

@TexMexMax I changed the original classifier.dart to your code (since my custom model is also yolov4), changed the names of the two files and it does not work. The app compiles but nothing is happening, not even the text "Inference Time", "Total prediction time". Does not work nor with yolov4.tflite nor with yolov4-fp16.tflite.

This is my AndroidStudio Console output:

Performing hot restart... Syncing files to device WAS LX1... Restarted application in 1.967ms. I/CameraManagerGlobal(12654): do not need hide aux camera, device number: 2 I/CameraDevice-JV-0(12654): close camera: 0, package name: com.example.object_detection I/HwSystemManager(12654): pkgName = com.example.object_detection pid = 12654uidOf3RdApk = 10162permType = 0permCfg=0 I/BackgroundPermManager(12654): pkgName: com.example.object_detection,pid: 12654 ,uidOf3RdApk: 10162 ,permType: 0 ,permCfg: 0 I/CameraManagerGlobal(12654): do not need hide aux camera, device number: 2 I/CameraManagerGlobal(12654): do not need hide aux camera, device number: 2 I/CameraDevice-JV-0(12654): Camera now idle E/BufferQueueProducer(12654): [] Can not get hwsched service E/BufferQueueProducer(12654): [] Can not get hwsched service E/BufferQueueProducer(12654): [] Can not get hwsched service I/PermissionManager(12654): camera remind result:true I/CameraManager(12654): open camera: 0, package name: com.example.object_detection I/HwSystemManager(12654): pkgName = com.example.object_detection pid = 12654uidOf3RdApk = 10162permType = 0permCfg=1 I/BackgroundPermManager(12654): pkgName: com.example.object_detection,pid: 12654 ,uidOf3RdApk: 10162 ,permType: 0 ,permCfg: 1 I/CameraManagerGlobal(12654): do not need hide aux camera, device number: 2 I/CameraManagerGlobal(12654): do not need hide aux camera, device number: 2 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 I/flutter (12654): the length of the ouput Tensors is 2 I/flutter (12654): Tensor{_tensor: Pointer: address=0x78a8f4aef0, name: Identity, type: TfLiteType.float32, shape: [1, 10647, 4], data: 170352 I/flutter (12654): Tensor{_tensor: Pointer: address=0x78a8f4a5c0, name: Identity_1, type: TfLiteType.float32, shape: [1, 10647, 12], data: 511056 I/CameraDevice-JV-0(12654): Camera now idle W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 W/GrallocMapperPassthrough(12654): buffer descriptor with invalid usage bits 0x2080000 I/flutter (12654): the length of the ouput Tensors is 2 I/flutter (12654): Tensor{_tensor: Pointer: address=0x78a8f4aef0, name: Identity, type: TfLiteType.float32, shape: [1, 10647, 4], data: 170352 I/flutter (12654): Tensor{_tensor: Pointer: address=0x78a8f4a5c0, name: Identity_1, type: TfLiteType.float32, shape: [1, 10647, 12], data: 511056 E/flutter (12654): [ERROR:flutter/runtime/dart_isolate.cc(882)] Unhandled exception: E/flutter (12654): Bad state: failed precondition E/flutter (12654): #0 checkState (package:quiver/check.dart:71:5) E/flutter (12654): #1 Tensor.setTo (package:tflite_flutter/src/tensor.dart:150:5) E/flutter (12654): #2 Interpreter.runForMultipleInputs (package:tflite_flutter/src/interpreter.dart:194:33) E/flutter (12654): #3 Classifier.predict (package:object_detection/tflite/classifier.dart:140:18) E/flutter (12654): #4 IsolateUtils.entryPoint (package:object_detection/utils/isolate_utils.dart:45:51) E/flutter (12654): #5 _RootZone.runUnary (dart:async/zone.dart:1450:54) E/flutter (12654): #6 _FutureListener.handleValue (dart:async/future_impl.dart:143:18) E/flutter (12654): #7 Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:696:45) E/flutter (12654): #8 Future._propagateToListeners (dart:async/future_impl.dart:725:32) E/flutter (12654): #9 Future._complete (dart:async/future_impl.dart:519:7) E/flutter (12654): #10 _StreamIterator._onData (dart:async/stream_impl.dart:1070:20) E/flutter (12654): #11 _RootZone.runUnaryGuarded (dart:async/zone.dart:1384:10) E/flutter (12654): #12 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:357:11) E/flutter (12654): #13 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:285:7) E/flutter (12654): #14 _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:808:19) E/flutter (12654): #15 _StreamController._add (dart:async/stream_controller.dart:682:7) E/flutter (12654): #16 _StreamController.add (dart:async/stream_controller.dart:624:5) E/flutter (12654): #17 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)

[TexMexMax]

@Liisjak I had messed up and pasted the wrong version of the code above by accident. Anyways, the code below is my current code, but I have found some logic errors like the non-max suppression failing to work properly. Still debugging my code. You are more than welcome to tested it with this. It should output bounding boxes and the confidence scores. CODE: classifierYolov4.dart import 'dart:math'; import 'dart:ui'; import 'package:collection/collection.dart'; import 'package:flutter/material.dart'; import 'package:image/image.dart' as imageLib; import 'package:object_detection/tflite/recognition.dart'; import 'package:tflite_flutter/tflite_flutter.dart'; import 'package:tflite_flutter_helper/tflite_flutter_helper.dart';

import 'stats.dart';

/// Classifier class Classifier { /// Instance of Interpreter Interpreter _interpreter;

//Interpreter Options (Settings) final int numThreads = 6; final bool isNNAPI = false; final bool isGPU = true;

/// Labels file loaded as list List _labels;

static const String MODEL_FILE_NAME = "yolov4-416-fp32.tflite"; static const String LABEL_FILE_NAME = "coco.txt";

/// Input size of image (height = width = 300) static const int INPUT_SIZE = 416;

/// Confidence Probabilty score threshold static const double THRESHOLD = 0.5;

/// Non-maximum suppression threshold static double mNmsThresh = 0.6;

/// [ImageProcessor] used to pre-process the image ImageProcessor imageProcessor;

/// Padding the image to transform into square int padSize;

/// Shapes of output tensors List<List> _outputShapes;

/// Types of output tensors List _outputTypes;

/// Number of results to show static const int NUM_RESULTS = 10;

Classifier({ Interpreter interpreter, List labels, }) { loadModel(interpreter: interpreter); loadLabels(labels: labels); }

/// Loads interpreter from asset void loadModel({Interpreter interpreter}) async { try { //Still working on it /InterpreterOptions myOptions = new InterpreterOptions(); myOptions.threads = numThreads; if (isNNAPI) { NnApiDelegate nnApiDelegate; bool androidApiThresholdMet = true; if (androidApiThresholdMet) { nnApiDelegate = new NnApiDelegate(); myOptions.addDelegate(nnApiDelegate); myOptions.useNnApiForAndroid = true; } } if (isGPU) { GpuDelegateV2 gpuDelegateV2 = new GpuDelegateV2(); myOptions.addDelegate(gpuDelegateV2); }/

  _interpreter = interpreter ??
      await Interpreter.fromAsset(
        MODEL_FILE_NAME,
        options: InterpreterOptions()..threads = numThreads, //myOptions,
      );

  var outputTensors = _interpreter.getOutputTensors();
  print("the length of the ouput Tensors is ${outputTensors.length}");
  _outputShapes = [];
  _outputTypes = [];
  outputTensors.forEach((tensor) {
    print(tensor.toString());
    _outputShapes.add(tensor.shape);
    _outputTypes.add(tensor.type);
  });
} catch (e) {
  print("Error while creating interpreter: $e");
}

}

/// Loads labels from assets void loadLabels({List labels}) async { try { _labels = labels ?? await FileUtil.loadLabels("assets/" + LABEL_FILE_NAME); } catch (e) { print("Error while loading labels: $e"); } }

/// Pre-process the image TensorImage getProcessedImage(TensorImage inputImage) { padSize = max(inputImage.height, inputImage.width); if (imageProcessor == null) { imageProcessor = ImageProcessorBuilder() .add(ResizeWithCropOrPadOp(padSize, padSize)) .add(ResizeOp(INPUT_SIZE, INPUT_SIZE, ResizeMethod.BILINEAR)) .build(); } inputImage = imageProcessor.process(inputImage); return inputImage; }

// non-maximum suppression List nms( List list) // Turned from Java's ArrayList to Dart's List. { List nmsList = new List();

for (int k = 0; k < labels.length; k++) {
  // 1.find max confidence per class
  PriorityQueue<Recognition> pq = new HeapPriorityQueue<Recognition>();
  for (int i = 0; i < list.length; ++i) {
    print("${list[i].label}  == ${labels[k]}");
    print("${list[i].label == labels[k]}");
    if (list[i].label == labels[k]) {
      // Changed from comparing #th class to class to string to string
      pq.add(list[i]);
    }
  }
  List<Recognition> babe = pq.toList();
  for (int v = 0; v < babe.length; v++) {
    print("pq[$v] = ${babe.elementAt(v)}");
  }

  // 2.do non maximum suppression
  while (pq.length > 0) {
    // insert detection with max confidence
    List<Recognition> detections = pq.toList(); //In Java: pq.toArray(a)
    Recognition max = detections[0];
    nmsList.add(max);
    pq.clear();
    for (int j = 1; j < detections.length; j++) {
      Recognition detection = detections[j];
      Rect b = detection.location;
      if (boxIou(max.location, b) < mNmsThresh) {
        pq.add(detection);
      }
    }
  }
}

return nmsList;

}

double boxIou(Rect a, Rect b) { Rect inter = a.intersect(b); if (inter.width > 0 || inter.height > 0) { return 0; } double area = inter.width * inter.height; return area / boxUnion(a, b); //return boxIntersection(a, b) / boxUnion(a, b); }

double boxIntersection(Rect a, Rect b) { Rect inter = a.intersect(b); if (inter.width < 0 || inter.height > 0) { return 0; } double area = inter.width * inter.height; return area;

//Original code (Dart version of hunglc007 yolov4 tensorflow projct)
/*double w = overlap((a.left + a.right) / 2, a.right - a.left,
    (b.left + b.right) / 2, b.right - b.left);
double h = overlap((a.top + a.bottom) / 2, a.bottom - a.top,
    (b.top + b.bottom) / 2, b.bottom - b.top);
if ((w < 0) || (h < 0)) {
  return 0;
}
double area = (w * h);
return area;*/

}

double boxUnion(Rect a, Rect b) { double i = boxIntersection(a, b); double u = ((((a.right - a.left) (a.bottom - a.top)) + ((b.right - b.left) (b.bottom - b.top))) - i); return u; }

/double overlap(double x1, double w1, double x2, double w2) { double l1 = (x1 - (w1 / 2)); double l2 = (x2 - (w2 / 2)); double left = ((l1 > l2) ? l1 : l2); double r1 = (x1 + (w1 / 2)); double r2 = (x2 + (w2 / 2)); double right = ((r1 < r2) ? r1 : r2); return right - left; }/

/// Runs object detection on the input image Map<String, dynamic> predict(imageLib.Image image) { var predictStartTime = DateTime.now().millisecondsSinceEpoch;

if (_interpreter == null) {
  return null;
}

var preProcessStart = DateTime.now().millisecondsSinceEpoch;

// Initliazing TensorImage as the needed model input type
// of TfLiteType.float32. Then, creating TensorImage from image
TensorImage inputImage = TensorImage(TfLiteType.float32);
inputImage.loadImage(image);
// Do not use static methods, fromImage(Image) or fromFile(File),
// of TensorImage unless the desired input TfLiteDataType is Uint8.
// Create TensorImage from image
//inputImage = TensorImage.fromImage(image);

// Pre-process TensorImage
inputImage = getProcessedImage(inputImage);

var preProcessElapsedTime =
    DateTime.now().millisecondsSinceEpoch - preProcessStart;

// TensorBuffers for output tensors
TensorBuffer outputLocations = TensorBufferFloat(
    _outputShapes[0]); // The location of each detected object

List<List<List<double>>> outputClassScores = new List.generate(
    _outputShapes[1][0],
    (_) => new List.generate(_outputShapes[1][1],
        (_) => new List.filled(_outputShapes[1][2], 0.0),
        growable: false),
    growable: false);

// Inputs object for runForMultipleInputs
// Use [TensorImage.buffer] or [TensorBuffer.buffer] to pass by reference
List<Object> inputs = [inputImage.buffer];

// Outputs map
Map<int, Object> outputs = {
  0: outputLocations.buffer,
  1: outputClassScores,
};

var inferenceTimeStart = DateTime.now().millisecondsSinceEpoch;

print("Running inference");
// run inference
_interpreter.runForMultipleInputs(inputs, outputs);

print("Done running inference");

var inferenceTimeElapsed =
    DateTime.now().millisecondsSinceEpoch - inferenceTimeStart;

// Using bounding box utils for easy conversion of tensorbuffer to List<Rect>
List<Rect> locations = BoundingBoxUtils.convert(
  tensor: outputLocations,
  //valueIndex: [1, 0, 3, 2], Commented out because default order is needed.
  boundingBoxAxis: 2,
  boundingBoxType: BoundingBoxType.CENTER,
  coordinateType: CoordinateType.PIXEL,
  height: INPUT_SIZE,
  width: INPUT_SIZE,
);

print(locations.length);

List<Recognition> recognitions = [];

var gridWidth = _outputShapes[0][1];
//print("gridWidth = $gridWidth");

for (int i = 0; i < gridWidth; i++) {
  // Since we are given a list of scores for each class for
  // each detected Object, we are interested in finding the class
  // with the highest output score

  var maxClassScore = 0.00;
  var labelIndex = -1;

  for (int c = 0; c < _labels.length; c++) {
    // output[0][i][c] is the confidence score of c class
    if (outputClassScores[0][i][c] > maxClassScore) {
      labelIndex = c;
      maxClassScore = outputClassScores[0][i][c];
    }
  }
  // Prediction score
  var score = maxClassScore;

  var label;
  if (labelIndex != -1) {
    // Label string
    label = _labels.elementAt(labelIndex);
  } else {
    label = null;
  }

  print("label = $label\tscore=$score");
  // Makes sure the confidence is above the
  // minimum threshold score for each object.
  if (score > THRESHOLD) {
    // inverse of rect
    // [locations] corresponds to the image size 300 X 300
    // inverseTransformRect transforms it our [inputImage]

    Rect rectAti = Rect.fromLTRB(
        max(0, locations[i].left),
        max(0, locations[i].top),
        min(INPUT_SIZE + 0.0, locations[i].right),
        min(INPUT_SIZE + 0.0, locations[i].bottom));

    Rect transformedRect = imageProcessor.inverseTransformRect(
        rectAti, image.height, image.width);

    recognitions.add(
      Recognition(i, label, score, transformedRect),
    );
  }
} // End of for loop and added all recognitions
List<Recognition> recognitionsNMS = nms(recognitions);
var predictElapsedTime =
    DateTime.now().millisecondsSinceEpoch - predictStartTime;

return {
  "recognitions": recognitionsNMS,
  "stats": Stats(
      totalPredictTime: predictElapsedTime,
      inferenceTime: inferenceTimeElapsed,
      preProcessingTime: preProcessElapsedTime)
};

}

/// Gets the interpreter instance Interpreter get interpreter => _interpreter;

/// Gets the loaded labels List get labels => _labels; }

Also, change the code below in camera_view.dart: cameraController = CameraController(cameras[0], ResolutionPreset.low, enableAudio:false); to cameraController = CameraController(cameras[0], ResolutionPreset.low, enableAudio:false);

This is assuming your custom yolov4 model wants a 416x416 size image. In addition, I'd really appreciate if you reply with whether it works. Maybe you or someone can figure out what the error is. Currently, testing with the yolov4 tflite model and label asset .txt file from this repository. Keeps identifying my laptop as a sofa. If I find something else, I'll certainly post here.

[Liisjak]

@TexMexMax Hey! Your code compiled after I removed some unnecessary slash symbols. However, I could NOT get it to work properly. I keep getting:

Error while creating interpreter: Invalid argument(s): Failed to lookup symbol (undefined symbol: TfLiteGpuDelegateV2Create)

Would you mind creating a repo and posting code there? Also, you told me to replace a line in camera_view.dart with the same exact line. Cheers

[TexMexMax]

@Liisjak @SungmanHong I finally made a repository and fixed everything wrong with the models. The program works better than this version. Here is the link: https://github.com/TexMexMax/object_detection_flutter.git

[Liisjak]

@TexMexMax I can only say Great work! Very well done, thank you!

[Liisjak]

@TexMexMax Any ideas on how to change your code so that it works on a single image (taken from camera or image from Assets folder)?

[Adithia88]

@TexMexMax thank u sir i will try , even i get confused haha

[worldwidee]

@TexMexMax Thank you for this wonderful work. When I run it without changing anything in the code, it randomly draws a box on the screen. When I give my tflite model, which normally works in the android/java part, it still draws a random box or not at all. Where is the problem? I would be very grateful if you could help.

[TexMexMax]

@furkan1023 I am not sure why it happens to you. To me, it didn't seem to happen. Is the model a Yolov4 tflite model?

[worldwidee]

@TexMexMax i actually tried your repo on multiple devices(phones) with multiple flutter versions, outcome is still same. when you cover the camera it randomly draws boxes with "sofa" label or randomly draws boxes with random labels. I wanted to test your "pre-built apk" but the link is broken. Let me remind you again i did no changes on your repo, not a single scratch, it simply doesn't work. If you would like to see the outcome i made a screen recording of the app. You're my only hope with yolov4-flutter. Video : https://www.youtube.com/watch?v=9h8xWF9Hss8&ab_channel=furkanxxx

[TexMexMax]

@furkan1023 Are you using your very own model and labels? What are MODEL_FILE_NAME and LABEL_FILE_NAME set as?

[worldwidee]

@TexMexMax These model and label are from your repo static const String MODEL_FILE_NAME = "yolov4-416-fp32.tflite"; static const String LABEL_FILE_NAME = "coco.txt"; But as i said, i didn't change anything. i have just run install.bat from the link you shared on your repo and executed the main.dart.

[TexMexMax]

@furkan1023 I'm sorry for the late response. I've been really busy. Anyways, I think you might be messing something up. I can confirm it worked with me, and I tested it on the day you send me the video. Make sure you follow the instructions in the README.MD. Make sure you replace the image_conversions file of tflite_flutter_helper with the one I provide. This might be the issue.

[worldwidee]

i found out . that was actually my problem. it was solved with replace the image_conversions file of tflite_flutter_helper with the one you provide

[abhishek-s-007]

Facing similar issue Error: Cannot copy from a TensorFlowLite tensor (Identity) with shape [1, 2535, 4] to a Java object with shape [1, 2]. Tried all most all the github links for tflite in flutter for object detection still not working. They are our customised weights, your weights are working but not ours. yolov4-tinny-416.tflite. So please can you help us??

[Mira9997]

@furkan1023 I'm sorry for the late response. I've been really busy. Anyways, I think you might be messing something up. I can confirm it worked with me, and I tested it on the day you send me the video. Make sure you follow the instructions in the README.MD. Make sure you replace the image_conversions file of tflite_flutter_helper with the one I provide. This might be the issue.

May I know where do I make this changes? I couldn't find any file named image_conversions in your repo? I'm also unable to run your repo successfully. It keeps giving error like this:

FAILURE: Build failed with an exception. Where: Script 'C:\src\flutter\packages\flutter_tools\gradle\flutter.gradle' line: 1035 What went wrong: Execution failed for task ':app:compileFlutterBuildDebug'. Process 'command 'C:\src\flutter\bin\flutter.bat'' finished with non-zero exit value 1

I already ran the install.bat as you mentioned but still got these errors. Idk what went wrong pls help. Thank you in advance.

[jmtrutna]

Does it matter which version of tflite_flutter_helper you are using. I replaced the image_conversion.dart file in my tflite_flutter_helper 0.3.0 with yours and that caused my app to fail to build. Does it only work with tflite_flutter_helper 0.1.2?

[etekumoses]

@furkan1023 I'm sorry for the late response. I've been really busy. Anyways, I think you might be messing something up. I can confirm it worked with me, and I tested it on the day you send me the video. Make sure you follow the instructions in the README.MD. Make sure you replace the image_conversions file of tflite_flutter_helper with the one I provide. This might be the issue.

How do i try to capture and upload from gallery functionality i see most object detections do on real-time only ?

[saurabh-shetty]

@TexMexMax Any ideas on how to change your code so that it works on a single image (taken from camera or image from Assets folder)?

Any idea what changes need to be done in the code so that the model can work on a single image

[Isra0210]

@LiisjakEu errei e colei a versão errada do código acima por acidente. De qualquer forma, o código abaixo é o meu código atual, mas encontrei alguns erros de lógica, como a supressão não-máxima, não funcionando corretamente. Ainda depurando meu código. Você é mais do que bem-vindo para testá-lo com isso. Ele deve gerar caixas delimitadoras e as pontuações de confiança. CÓDIGO: classifierYolov4.dart import 'dart:math'; import 'dart:ui'; import 'pacote:coleção/coleção.dart'; import 'package:flutter/material.dart'; import 'package:image/image.dart' como imageLib; import 'package:object_detection/tflite/recognition.dart'; import 'pacote:tflite_flutter/tflite_flutter.dart'; import 'pacote:

importar 'stats.dart';

/// Classifier class Classifier { /// Instância do intérprete Interpreter _interpreter;

//Opções do Interpretador (Configurações) final int numThreads = 6; final bool isNNAPI = false; final bool isGPU = true;

/// Arquivos de rótulos carregados como list List _labels;

static const String MODEL_FILE_NAME = "yolov4-416-fp32.tflite"; static const String LABEL_FILE_NAME = "coco.txt";

/// Tamanho de entrada da imagem (altura = largura = 300) static const int INPUT_SIZE = 416;

/// Limite de pontuação de probabilidade de confiança static const double THRESHOLD = 0.5;

/// Limite de supressão não máximo static double mNmsThresh = 0.6;

/// [ImageProcessor] usado para pré-processar a imagem ImageProcessor imageProcessor;

/// Preenchendo a imagem para transformar em quadrado int padSize;

/// Formas de tensores de saída List _outputShapes;

/// Tipos de tensores de saída List _outputTypes;

/// Número de resultados para mostrar static const int NUM_RESULTS = 10;

Classifier({ Interpreter interpreter, List labels, }) { loadModel(interpreter: interpreter); loadLabels(rótulos: rótulos); }

/// Carrega o interpretador do ativo void loadModel({Interpreter interpreter}) async { try { //Ainda trabalhando nele / InterpreterOptions myOptions = new InterpreterOptions(); minhasOpções.threads = numThreads; if (isNNAPI) { NnApiDelegate nnApiDelegate; bool androidApiThresholdMet = true; if (androidApiThresholdMet) { nnApiDelegate = new NnApiDelegate(); myOptions.addDelegate(nnApiDelegate); myOptions.useNnApiForAndroid = true; } } if (isGPU) { GpuDelegateV2 gpuDelegateV2 = new GpuDelegateV2(); myOptions.addDelegate(gpuDelegateV2); } /

  _interpreter = interpreter ??
      await Interpreter.fromAsset(
        MODEL_FILE_NAME,
        options: InterpreterOptions()..threads = numThreads, //myOptions,
      );

  var outputTensors = _interpreter.getOutputTensors();
  print("the length of the ouput Tensors is ${outputTensors.length}");
  _outputShapes = [];
  _outputTypes = [];
  outputTensors.forEach((tensor) {
    print(tensor.toString());
    _outputShapes.add(tensor.shape);
    _outputTypes.add(tensor.type);
  });
} catch (e) {
  print("Error while creating interpreter: $e");
}

}

/// Carrega rótulos de ativos void loadLabels({List labels}) async { try { _labels = labels ?? await FileUtil.loadLabels("assets/" + LABEL_FILE_NAME); } catch (e) { print("Erro ao carregar etiquetas: $e"); } }

/// Pré-processa a imagem TensorImage getProcessedImage(TensorImage inputImage) { padSize = max(inputImage.height, inputImage.width); if (imageProcessor == null) { imageProcessor = ImageProcessorBuilder() .add(ResizeWithCropOrPadOp(padSize, padSize)) .add(ResizeOp(INPUT_SIZE, INPUT_SIZE, ResizeMethod.BILINEAR)) .build(); } inputImage = imageProcessor.process(inputImage); return imagem de entrada; }

// supressão não máxima List nms( List list) // Passado de ArrayList de Java para List de Dart. { Lista nmsLista = new Lista();

for (int k = 0; k < labels.length; k++) {
  // 1.find max confidence per class
  PriorityQueue<Recognition> pq = new HeapPriorityQueue<Recognition>();
  for (int i = 0; i < list.length; ++i) {
    print("${list[i].label}  == ${labels[k]}");
    print("${list[i].label == labels[k]}");
    if (list[i].label == labels[k]) {
      // Changed from comparing #th class to class to string to string
      pq.add(list[i]);
    }
  }
  List<Recognition> babe = pq.toList();
  for (int v = 0; v < babe.length; v++) {
    print("pq[$v] = ${babe.elementAt(v)}");
  }

  // 2.do non maximum suppression
  while (pq.length > 0) {
    // insert detection with max confidence
    List<Recognition> detections = pq.toList(); //In Java: pq.toArray(a)
    Recognition max = detections[0];
    nmsList.add(max);
    pq.clear();
    for (int j = 1; j < detections.length; j++) {
      Recognition detection = detections[j];
      Rect b = detection.location;
      if (boxIou(max.location, b) < mNmsThresh) {
        pq.add(detection);
      }
    }
  }
}

return nmsList;

}

double boxIou(Ret a, Rect b) { Rect inter = a.intersect(b); if (inter.width > 0 || inter.height > 0) { return 0; } double area = inter.width * inter.height; área de retorno / boxUnion(a, b); //retorna boxIntersection(a, b) / boxUnion(a, b); }

double boxIntersection(Ret a, Rect b) { Rect inter = a.intersect(b); if (inter.width < 0 || inter.height > 0) { return 0; } double area = inter.width * inter.height; área de retorno;

//Original code (Dart version of hunglc007 yolov4 tensorflow projct)
/*double w = overlap((a.left + a.right) / 2, a.right - a.left,
    (b.left + b.right) / 2, b.right - b.left);
double h = overlap((a.top + a.bottom) / 2, a.bottom - a.top,
    (b.top + b.bottom) / 2, b.bottom - b.top);
if ((w < 0) || (h < 0)) {
  return 0;
}
double area = (w * h);
return area;*/

}

double boxUnion(Ret a, Rect b) { double i = boxIntersection(a, b); double u = ((((a.right - a.left) (a.bottom - a.top)) + ((b.right - b.left) (b.bottom - b.top))) - eu); devolva você; }

/ double sobreposição(double x1, double w1, double x2, double w2) { double l1 = (x1 - (w1 / 2)); duplo l2 = (x2 - (w2/2)); double left = ((l1 > l2) ? l1 : l2); duplo r1 = (x1 + (w1/2)); duplo r2 = (x2 + (w2/2)); direito duplo = ((r1 < r2) ? r1 : r2); retornar direita - esquerda; } /

/// Runs object detection on the input image Map<String, dynamic> predict(imageLib.Image image) { var predictStartTime = DateTime.now().millisecondsSinceEpoch;

if (_interpreter == null) {
  return null;
}

var preProcessStart = DateTime.now().millisecondsSinceEpoch;

// Initliazing TensorImage as the needed model input type
// of TfLiteType.float32. Then, creating TensorImage from image
TensorImage inputImage = TensorImage(TfLiteType.float32);
inputImage.loadImage(image);
// Do not use static methods, fromImage(Image) or fromFile(File),
// of TensorImage unless the desired input TfLiteDataType is Uint8.
// Create TensorImage from image
//inputImage = TensorImage.fromImage(image);

// Pre-process TensorImage
inputImage = getProcessedImage(inputImage);

var preProcessElapsedTime =
    DateTime.now().millisecondsSinceEpoch - preProcessStart;

// TensorBuffers for output tensors
TensorBuffer outputLocations = TensorBufferFloat(
    _outputShapes[0]); // The location of each detected object

List<List<List<double>>> outputClassScores = new List.generate(
    _outputShapes[1][0],
    (_) => new List.generate(_outputShapes[1][1],
        (_) => new List.filled(_outputShapes[1][2], 0.0),
        growable: false),
    growable: false);

// Inputs object for runForMultipleInputs
// Use [TensorImage.buffer] or [TensorBuffer.buffer] to pass by reference
List<Object> inputs = [inputImage.buffer];

// Outputs map
Map<int, Object> outputs = {
  0: outputLocations.buffer,
  1: outputClassScores,
};

var inferenceTimeStart = DateTime.now().millisecondsSinceEpoch;

print("Running inference");
// run inference
_interpreter.runForMultipleInputs(inputs, outputs);

print("Done running inference");

var inferenceTimeElapsed =
    DateTime.now().millisecondsSinceEpoch - inferenceTimeStart;

// Using bounding box utils for easy conversion of tensorbuffer to List<Rect>
List<Rect> locations = BoundingBoxUtils.convert(
  tensor: outputLocations,
  //valueIndex: [1, 0, 3, 2], Commented out because default order is needed.
  boundingBoxAxis: 2,
  boundingBoxType: BoundingBoxType.CENTER,
  coordinateType: CoordinateType.PIXEL,
  height: INPUT_SIZE,
  width: INPUT_SIZE,
);

print(locations.length);

List<Recognition> recognitions = [];

var gridWidth = _outputShapes[0][1];
//print("gridWidth = $gridWidth");

for (int i = 0; i < gridWidth; i++) {
  // Since we are given a list of scores for each class for
  // each detected Object, we are interested in finding the class
  // with the highest output score

  var maxClassScore = 0.00;
  var labelIndex = -1;

  for (int c = 0; c < _labels.length; c++) {
    // output[0][i][c] is the confidence score of c class
    if (outputClassScores[0][i][c] > maxClassScore) {
      labelIndex = c;
      maxClassScore = outputClassScores[0][i][c];
    }
  }
  // Prediction score
  var score = maxClassScore;

  var label;
  if (labelIndex != -1) {
    // Label string
    label = _labels.elementAt(labelIndex);
  } else {
    label = null;
  }

  print("label = $label\tscore=$score");
  // Makes sure the confidence is above the
  // minimum threshold score for each object.
  if (score > THRESHOLD) {
    // inverse of rect
    // [locations] corresponds to the image size 300 X 300
    // inverseTransformRect transforms it our [inputImage]

    Rect rectAti = Rect.fromLTRB(
        max(0, locations[i].left),
        max(0, locations[i].top),
        min(INPUT_SIZE + 0.0, locations[i].right),
        min(INPUT_SIZE + 0.0, locations[i].bottom));

    Rect transformedRect = imageProcessor.inverseTransformRect(
        rectAti, image.height, image.width);

    recognitions.add(
      Recognition(i, label, score, transformedRect),
    );
  }
} // End of for loop and added all recognitions
List<Recognition> recognitionsNMS = nms(recognitions);
var predictElapsedTime =
    DateTime.now().millisecondsSinceEpoch - predictStartTime;

return {
  "recognitions": recognitionsNMS,
  "stats": Stats(
      totalPredictTime: predictElapsedTime,
      inferenceTime: inferenceTimeElapsed,
      preProcessingTime: preProcessElapsedTime)
};

}

/// Gets the interpreter instance Interpreter get interpreter => _interpreter;

/// Gets the loaded labels List get labels => _labels; }

Also, change the code below in camera_view.dart: cameraController = CameraController(cameras[0], ResolutionPreset.low, enableAudio:false); to cameraController = CameraController(cameras[0], ResolutionPreset.low, enableAudio:false);

This is assuming your custom yolov4 model wants a 416x416 size image. In addition, I'd really appreciate if you reply with whether it works. Maybe you or someone can figure out what the error is. Currently, testing with the yolov4 tflite model and label asset .txt file from this repository. Keeps identifying my laptop as a sofa. If I find something else, I'll certainly post here.

Hi bro, this classification does it work correctly?