TFLiteSwift-Vision

Table

• Goal
• Requirements
• Usage
• Getting Started with Example
• Architecture
• Done and TODO
• License

Goal

This library is a TensorFlowLiteSwift wrapping library for vision pre/post-processing. You can use TFLiteSwift-Vision, if you want to implemented preprocessing and postprocessing functions in the repository.

Here is more detail of this repo's background and goal, and what is diff.

Requirements

• Xcode
• CocodaPods
• iOS 10.0+

Usage

Install the TFLiteSwift-Vision

TFLiteSwift-Vision is available through CocoaPods. To install it, simply add the following line to your Podfile:

target 'MyXcodeProject' do
  use_frameworks!

  # Pods for Your Project
  pod 'TFLiteSwift-Vision', '~> 0.2.6'

end

post_install do |installer|
  installer.pods_project.build_configurations.each do |config|
    config.build_settings["EXCLUDED_ARCHS[sdk=iphonesimulator*]"] = "arm64"
  end
end

And then, run following:

pod install

Add .tflite file in the Xcode Project

If you have other label file or other meta data file, also import it.

Implement Inference Code

Import TFLiteSwift_Vision framework.

import TFLiteSwift_Vision

Setup interpreter.

let options = TFLiteVisionInterpreter.Options(
  modelName: "mobilenet_v2_1.0_224",
  inputRankType: .bwhc, // if it is pytorch model, use `.bchw`
  normalization: .scaled(from: 0.0, to: 1.0)
)
var visionInterpreter = try? TFLiteVisionInterpreter(options: options)

Inference with an image. The following is an image classification case.

// inference
guard let output: TFLiteFlatArray = try? self.visionInterpreter?.inference(with: uiImage)?.first
    else { fatalError("Cannot inference") }

// postprocess
let predictedIndex: Int = Int(output.argmax())
print("predicted index: \(predictedLabel)")
print(output.dimensions)

Getting Started with Example

Clone and open the Example project

git clone https://github.com/tucan9389/TFLiteSwift-Vision
cd TFLiteSwift-Vision/Example
pod install
open TFLiteSwift-Vision.xcworkspace

Download model and label file

Download tflite model and label txt, and then import the files into Xcode project.

You can also download the following files on here

• mobilenet_v2_1.0_224.tflite
• labels_mobilenet_quant_v1_224.txt

Build and Run

After build and run the project, you can test the model(mobilenet_v2_1.0_224.tflite) with your own image data.

image classification

Architecture

Done and TODO

TFLiteSwift-Vision is supporting (or wants to support) follow functions:

• Preprocessing (convert Cocoa's image type to TFLiteSwift's Tensor)
  • Supporting input image type:
  • [x] UIImage
  • [x] CVPixelBuffer
  • [ ] CGImage
  • [ ] MTLTexture
  • Supporing bhwc sequence:
  • [x] bwhc (normally used in TF)
  • [x] bcwh (noramlly used in Pytorch)
  • [x] bwh (for gray image)
  • [x] bhw (for gray image)
  • [ ] whc
  • [ ] hwc
  • [ ] wh (for gray input)
  • Supporting normalization methods:
  • [x] Normalization with scaling (0...255 → 0.0...1.0)
  • [x] Normalization with mean and std (normaly used in pytorch and it is used in ImageNet firstly)
  • [x] Grayscaling (not to 4 dim tensor, but to 3 dim tensor from an image)
  • Supporting cropping methods:
  • [x] Resizing (vImageScale_ARGB8888)
  • [x] Centercropping
  • [ ] Padding
  • If basic functions are implemented, need to optimize with Metal or Accelerate (or other domain specific frameworks)
  • Supporting input type
  • [x] Float32
  • [x] UInt8
  • Supporting output type
  • [x] Float32
  • [x] UInt8
• Inferenceing
  • batch size
  • [x] 1 batch
  • [ ] n batch
  • [x] cpu or gpu(metal) selectable
• Postprocessing (convert TFLiteSwift's Tensor to Cocoa's type)
  • [ ] Tensor → UIImage
  • [ ] Tensor → MTLTexture
• Domain specific postprocessing examples
  • [x] Image classification
  • [ ] Object detection
  • [ ] Semantic segmentation
  • [ ] Pose estimation
• Replace TensorFlowLiteSwift to TFLiteSwift-Vision in tensorflow/examples
  • [x] image_classification
  • [x] object_detection
  • [x] posenet
  • [x] digit_classification
  • [x] semantic_segmentation

Author

• @tucan9389, tucan.dev@gmail.com
• @Seonghun23, kimsh777kr@gmail.com

License

TFLiteSwift-Vision is available under the Apache license. See the LICENSE file for more info.