Does it support YOLO5 and YOLO8 ?

[maheralzoubi97]

// ignore_for_file: avoid_print, depend_on_referenced_packages, unnecessary_null_comparison

import 'dart:math'; import 'dart:ui';

import 'package:flutter/material.dart'; import 'package:image/image.dart' as image_lib; import 'package:tflite_flutter/tflite_flutter.dart'; import 'package:tflite_flutter_helper/tflite_flutter_helper.dart'; import '/core/ml/realtime_object_detection_classifier/recognition.dart';

import 'stats.dart';

/// Classifier class Classifier { /// Instance of Interpreter late Interpreter _interpreter;

/// Labels file loaded as list late List _labels;

static const String modelFileName = "detect.tflite"; static const String labelFileName = "labelmap.txt";

/// Input size of image (height = width = 300) static const int inputSize = 300;

/// Result score threshold static const double threshold = 0.5;

/// [ImageProcessor] used to pre-process the image ImageProcessor? imageProcessor;

/// Padding the image to transform into square late int padSize;

/// Shapes of output tensors late List<List> _outputShapes;

/// Types of output tensors late List _outputTypes;

/// Number of results to show static const int numResult = 10;

Classifier({ Interpreter? interpreter, List? labels, }) { loadModel(interpreter: interpreter); loadLabels(labels: labels); }

/// Loads interpreter from asset void loadModel({Interpreter? interpreter}) async { try { _interpreter = interpreter ?? await Interpreter.fromAsset( modelFileName, options: InterpreterOptions()..threads = 4, );

  var outputTensors = _interpreter.getOutputTensors();
  _outputShapes = [];
  _outputTypes = [];
  for (var tensor in outputTensors) {
    _outputShapes.add(tensor.shape);
    _outputTypes.add(tensor.type);
  }
} catch (e) {
  print("Error while creating interpreter: $e");
}

}

/// Loads labels from assets void loadLabels({List? labels}) async { try { _labels = labels ?? await FileUtil.loadLabels("assets/$labelFileName"); } catch (e) { print("Error while loading labels: $e"); } }

/// Pre-process the image TensorImage getProcessedImage(TensorImage inputImage) { padSize = max(inputImage.height, inputImage.width); imageProcessor ??= ImageProcessorBuilder() .add(ResizeWithCropOrPadOp(padSize, padSize)) .add(ResizeOp(inputSize, inputSize, ResizeMethod.BILINEAR)) .build(); inputImage = imageProcessor!.process(inputImage); return inputImage; }

/// Runs object detection on the input image Map<String, dynamic>? predict(image_lib.Image image) { var predictStartTime = DateTime.now().millisecondsSinceEpoch;

if (_interpreter == null) {
  print("Interpreter not initialized");
  return null;
}

var preProcessStart = DateTime.now().millisecondsSinceEpoch;

// Create TensorImage from image
TensorImage inputImage = TensorImage.fromImage(image);

// Pre-process TensorImage
inputImage = getProcessedImage(inputImage);

var preProcessElapsedTime =
    DateTime.now().millisecondsSinceEpoch - preProcessStart;

// TensorBuffers for output tensors
TensorBuffer outputLocations = TensorBufferFloat(_outputShapes[0]);
TensorBuffer outputClasses = TensorBufferFloat(_outputShapes[1]);
TensorBuffer outputScores = TensorBufferFloat(_outputShapes[2]);
TensorBuffer numLocations = TensorBufferFloat(_outputShapes[3]);

// Inputs object for runForMultipleInputs
// Use [TensorImage.buffer] or [TensorBuffer.buffer] to pass by reference
List<Object> inputs = [inputImage.buffer];

// Outputs map
Map<int, Object> outputs = {
  0: outputLocations.buffer,
  1: outputClasses.buffer,
  2: outputScores.buffer,
  3: numLocations.buffer,
};

var inferenceTimeStart = DateTime.now().millisecondsSinceEpoch;

// run inference
_interpreter.runForMultipleInputs(inputs, outputs);

var inferenceTimeElapsed =
    DateTime.now().millisecondsSinceEpoch - inferenceTimeStart;

// Maximum number of results to show
int resultsCount = min(numResult, numLocations.getIntValue(0));

// Using labelOffset = 1 as ??? at index 0
int labelOffset = 1;

// Using bounding box utils for easy conversion of tensorbuffer to List<Rect>
List<Rect> locations = BoundingBoxUtils.convert(
  tensor: outputLocations,
  valueIndex: [1, 0, 3, 2],
  boundingBoxAxis: 2,
  boundingBoxType: BoundingBoxType.BOUNDARIES,
  coordinateType: CoordinateType.RATIO,
  height: inputSize,
  width: inputSize,
);

List<Recognition> recognitions = [];

for (int i = 0; i < resultsCount; i++) {
  // Prediction score
  var score = outputScores.getDoubleValue(i);

  // Label string
  var labelIndex = outputClasses.getIntValue(i) + labelOffset;
  var label = _labels.elementAt(labelIndex);

  if (score > threshold) {
    // inverse of rect
    // [locations] corresponds to the image size 300 X 300
    // inverseTransformRect transforms it our [inputImage]
    Rect transformedRect = imageProcessor!.inverseTransformRect(
        locations[i], image.height, image.width);

    recognitions.add(
      Recognition(i, label, score, transformedRect),
    );
  }
}

var predictElapsedTime =
    DateTime.now().millisecondsSinceEpoch - predictStartTime;

return {
  "recognitions": recognitions,
  "stats": Stats(
      totalPredictTime: predictElapsedTime,
      inferenceTime: inferenceTimeElapsed,
      preProcessingTime: preProcessElapsedTime)
};

}

/// Gets the interpreter instance Interpreter get interpreter => _interpreter;

/// Gets the loaded labels List get labels => _labels; }

when change modelFileName return error range index 0:1 @jaeyounkim @densa

[codscino]

I have not understood your example, but here is a working repo with yolov8 https://github.com/codscino/yolo_tflite

[maheralzoubi97]

thanks , but the example not real time object detection ? i need code working real time object detection @codscino

[codscino]

I was working on the same thing, but I did not find any time this week. Here are the suggestions to make it work https://github.com/tensorflow/flutter-tflite/pull/107#issuecomment-1658844538 . Please share if you manage to make it working

[VadymPinchuk]

@maheralzoubi97 Please describe your question better, as from the code example, taken from live_object_detectionssd, not really gives an understanding of your question. Also, please read Readme.md file, where it is mentioned that models are downloaded from the remote storage. In case of YOLO - that might not work, but I am not an ML specialist to help you here.

[codscino]

The remote download it is not a problem. I think just putting the tflite model in assets/models folder is ok

[iammohit1311]

I was working on the same thing, but I did not find any time this week. Here are the suggestions to make it work #107 (comment) . Please share if you manage to make it working

Let's work together on implementing live object detection, if you find time. I'm up for it!

[codscino]

I was working on the same thing, but I did not find any time this week. Here are the suggestions to make it work #107 (comment) . Please share if you manage to make it working

Let's work together on implementing live object detection, if you find time. I'm up for it!

Hi, that is a good idea, unfortunately I am very busy till the 10th of September. Then I will have some free time to work on this.

[ferraridamiano]

Here are the suggestions to make it work #107 (comment) .

To complete the answer you mentioned I would like to add the following. YOLO returns all the candidate bounding boxes (shape of the output: 1, 84, 8400). This means that as an output we get 8400 bounding boxes. But we then need to filter them out with non maxima suppression. This means that in order to get some useful box we need to write a dart implementation of the NMS algorithm

[codscino]

@ferraridamiano thanks for the feedback. This seems longer than what I expected. Are you working on this as well?

[ferraridamiano]

@codscino I'm interested but I'm not working on it yet. I think that it would not be that hard to do

[codscino]

Ok thanks for the feedback. When I will start working on it on September I will let you know

[UlisesGallardo]

I am also working on using yolov5 in a Flutter application, as @ferraridamiano mentions it is necessary to have the NMS, but I just found a Chinese forum where a person tries to implement yolov5 with a different apporach:

forum: https://qiita.com/syu-kwsk/items/e3126f55895444aa408b repo: https://github.com/syu-kwsk/flutter_yolov5_app

I'm looking at a way to implement it using the latest version of flutter-tflite

[iammohit1311]

I am also working on using yolov5 in a Flutter application, as @ferraridamiano mentions it is necessary to have the NMS, but I just found a Chinese forum where a person tries to implement yolov5 with a different apporach:

forum: https://qiita.com/syu-kwsk/items/e3126f55895444aa408b repo: https://github.com/syu-kwsk/flutter_yolov5_app

I'm looking at a way to implement it using the latest version of flutter-tflite

Were you able to make YOLOv5 model work in Flutter app using this repo ? Will try running my model using this repo. Thanks!

[ferraridamiano]

I successfully implemented a simple NMS algorithm in dart. I still have to optimize it because it's not that fast, but it works. I will release everything as open source as soon as I'm satisfied with it (maybe alongside a dart package for NMS). I'm happy that someone is working on it with different approches.

A side note: I noticed that the inference time (without NMS) of yolov8n (fp16) is really slow (~1600 ms on Snapdragon 730), while the official ultralytics app is like 10x faster. Are you experiencing similar behaviour?

[UlisesGallardo]

I am also working on using yolov5 in a Flutter application, as @ferraridamiano mentions it is necessary to have the NMS, but I just found a Chinese forum where a person tries to implement yolov5 with a different apporach: forum: https://qiita.com/syu-kwsk/items/e3126f55895444aa408b repo: https://github.com/syu-kwsk/flutter_yolov5_app I'm looking at a way to implement it using the latest version of flutter-tflite

Were you able to make YOLOv5 model work in Flutter app using this repo ? Will try running my model using this repo. Thanks!

I decided not use the code from the forum because he uses the package tflite_flutter_helper which it is right now discontinued, and I would like to use the new development that is not released. But you can try to implement it. Meanwhile I'm using the pytorch_flutter library to experiment with real time object detection using yolov5.

[codscino]

I am also working on using yolov5 in a Flutter application, as @ferraridamiano mentions it is necessary to have the NMS, but I just found a Chinese forum where a person tries to implement yolov5 with a different apporach:

forum: https://qiita.com/syu-kwsk/items/e3126f55895444aa408b repo: https://github.com/syu-kwsk/flutter_yolov5_app

I'm looking at a way to implement it using the latest version of flutter-tflite

Were you able to make YOLOv5 model work in Flutter app using this repo ? Will try running my model using this repo. Thanks!

I decided not use the code from the forum because he uses the package tflite_flutter_helper which it is right now discontinued, and I would like to use the new development that is not released. But you can try to implement it. Meanwhile I'm using the pytorch_flutter library to experiment with real time object detection using yolov5.

What is your inference time with yolov8n with the PyTorch package?

[codscino]

I successfully implemented a simple NMS algorithm in dart. I still have to optimize it because it's not that fast, but it works. I will release everything as open source as soon as I'm satisfied with it (maybe alongside a dart package for NMS). I'm happy that someone is working on it with different approches.

A side note: I noticed that the inference time (without NMS) of yolov8n (fp16) is really slow (~1600 ms on Snapdragon 730), while the official ultralytics app is like 10x faster. Are you experiencing similar behaviour?

Thanks a lot. I do not know, I have asked on their discord but currently they have just a paid mobile sdk for yolo. I think they are using flutter in their app because I have seen a flutter dev job listing in their career page. This gives hope but I have not the experience to achieve such performances.

[tnghieu]

I have not understood your example, but here is a working repo with yolov8 https://github.com/codscino/yolo_tflite

I'm new to ML. How do I get the output detections from the interpreter with your example?

[codscino]

I have not understood your example, but here is a working repo with yolov8 https://github.com/codscino/yolo_tflite

I'm new to ML. How do I get the output detections from the interpreter with your example?

I don't know sorry, it is a very basic example

[UlisesGallardo]

I am also working on using yolov5 in a Flutter application, as @ferraridamiano mentions it is necessary to have the NMS, but I just found a Chinese forum where a person tries to implement yolov5 with a different apporach:

forum: https://qiita.com/syu-kwsk/items/e3126f55895444aa408b repo: https://github.com/syu-kwsk/flutter_yolov5_app

I'm looking at a way to implement it using the latest version of flutter-tflite

Were you able to make YOLOv5 model work in Flutter app using this repo ? Will try running my model using this repo. Thanks!

I decided not use the code from the forum because he uses the package tflite_flutter_helper which it is right now discontinued, and I would like to use the new development that is not released. But you can try to implement it. Meanwhile I'm using the pytorch_flutter library to experiment with real time object detection using yolov5.

What is your inference time with yolov8n with the PyTorch package?

I'm using a fine tune yolov5s model in a custom dataset, and this is how I meassure the inference in flutter:
stopwatch.start(); List<ResultObjectDetection?> objDetect = await _objectModel.getImagePredictionList(bytes.first, minimumScore: 0.3, IOUThershold: 0.4); stopwatch.stop(); setState(() { _timeInference = stopwatch.elapsedMilliseconds; }); stopwatch.reset(); So, with a redmi note 9, it runs low with ~1200 ms under stream images. I'm not really sure if I'm doing something wrong.

[datka134]

I am also working on using yolov5 in a Flutter application, as @ferraridamiano mentions it is necessary to have the NMS, but I just found a Chinese forum where a person tries to implement yolov5 with a different apporach:

forum: https://qiita.com/syu-kwsk/items/e3126f55895444aa408b repo: https://github.com/syu-kwsk/flutter_yolov5_app

I'm looking at a way to implement it using the latest version of flutter-tflite

Were you able to make YOLOv5 model work in Flutter app using this repo ? Will try running my model using this repo. Thanks!

I decided not use the code from the forum because he uses the package tflite_flutter_helper which it is right now discontinued, and I would like to use the new development that is not released. But you can try to implement it. Meanwhile I'm using the pytorch_flutter library to experiment with real time object detection using yolov5.

What is your inference time with yolov8n with the PyTorch package?

I'm using a fine tune yolov5s model in a custom dataset, and this is how I meassure the inference in flutter: stopwatch.start(); List<ResultObjectDetection?> objDetect = await _objectModel.getImagePredictionList(bytes.first, minimumScore: 0.3, IOUThershold: 0.4); stopwatch.stop(); setState(() { _timeInference = stopwatch.elapsedMilliseconds; }); stopwatch.reset(); So, with a redmi note 9, it runs low with ~1200 ms under stream images. I'm not really sure if I'm doing something wrong.

i am working in deploying yolo on flutter too , but i think if you dont use platform channel nms algorithm is must have , im new to this too and right now im think of using pytorch on flutter than tflite . if u have any progress pls share it with me , thanks

[denesbartha]

Hi, any updates on this?

I am trying to implement nms in dart natively. I don't actually need the precise areas just an indication whether an object is present on the image or not.

[ferraridamiano]

Here I am. I just published my work as opensource here: https://github.com/ferraridamiano/yolo_flutter I really wanted to improve it but I do not have enough time to put on this. The project contains a simple (and not optimized) version of the NMS algorithm. Feel free to open PRs to improve the project

[tnghieu]

Here I am. I just published my work as opensource here: https://github.com/ferraridamiano/yolo_flutter

I really wanted to improve it but I do not have enough time to put on this. The project contains a simple (and not optimized) version of the NMS algorithm. Feel free to open PRs to improve the project

Awesome work! Did you export your yolo weights to tflite?

[ferraridamiano]

I don't want to include weights on the GitHub repo. But I wrote instructions on the README of the project that explains how to do it