DeepLab with TensorFlow Mobile or TensorFlow Lite

[dailystudio]

Hello @aquariusjay,

We just want to run this modal on Android. We have tried two approach TensorFlow Mobile and TensorFlow Lite.

With TensorFlow Mobile, we download the pre-trained modals with MobileNetV2: mobilenetv2_coco_voc_trainaug mobilenetv2_coco_voc_trainval mobilenetv2_coco_cityscapes_trainfine

We can successfully load the modal, but when run the inference, we get the following error:

05-16 16:06:34.122 611-635/? E/AndroidRuntime: FATAL EXCEPTION: AsyncTask #1
    Process: com.orange.labs.colorme.dev, PID: 611
    java.lang.RuntimeException: An error occurred while executing doInBackground()
        at android.os.AsyncTask$3.done(AsyncTask.java:325)
        at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:354)
        at java.util.concurrent.FutureTask.setException(FutureTask.java:223)
        at java.util.concurrent.FutureTask.run(FutureTask.java:242)
        at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:243)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
        at java.lang.Thread.run(Thread.java:761)
     Caused by: java.lang.IllegalArgumentException: No OpKernel was registered to support Op 'Slice' with these attrs.  Registered devices: [CPU], Registered kernels:
      device='CPU'; T in [DT_BOOL]
      device='CPU'; T in [DT_FLOAT]
      device='CPU'; T in [DT_INT32]

         [[Node: SemanticPredictions = Slice[Index=DT_INT32, T=DT_INT64](ArgMax, SemanticPredictions/begin, SemanticPredictions/size)]]
        at org.tensorflow.Session.run(Native Method)
        at org.tensorflow.Session.access$100(Session.java:48)
        at org.tensorflow.Session$Runner.runHelper(Session.java:298)
        at org.tensorflow.Session$Runner.runAndFetchMetadata(Session.java:260)
        at org.tensorflow.contrib.android.TensorFlowInferenceInterface.run(TensorFlowInferenceInterface.java:220)
        at org.tensorflow.contrib.android.TensorFlowInferenceInterface.run(TensorFlowInferenceInterface.java:197)
        at com.dailystudio.deeplab.DeeplabV3.segment(DeeplabV3.java:104)
        at com.dailystudio.deeplab.DeeplabApplication$1.doInBackground(DeeplabApplication.java:46)
        at com.dailystudio.deeplab.DeeplabApplication$1.doInBackground(DeeplabApplication.java:22)
        at android.os.AsyncTask$2.call(AsyncTask.java:305)
        at java.util.concurrent.FutureTask.run(FutureTask.java:237)

I think this is caused by the output node "SemanticPredictions" call the operation Slice with INT64 data. This is not supported by TensorFlow Mobile yet.

With TensorFlow Lite, we use the following command to convert is to tflite format:

toco \
    --input_file=$(pwd)/model/frozen_inference_graph.pb \
    --input_format=TENSORFLOW_GRAPHDEF \
    --output_format=TFLITE \
    --output_file=$(pwd)/model/deeplabv3_mnv2_pascal_trainval.tflite \
    --inference_type=FLOAT \
    --input_type=QUANTIZED_UINT8 \
    --input_arrays=ImageTensor \
    --output_arrays=SemanticPredictions \
    --input_shapes=1,513,513,3 \
    --default_ranges_min=0 --default_ranges_max=255

We get the following warnings:

2018-05-16 16:19:29.429205: W tensorflow/contrib/lite/toco/toco_cmdline_flags.cc:245] --input_type is deprecated. It was an ambiguous flag that set both --input_data_types and --inference_input_type. If you are trying to complement the input file with information about the type of input arrays, use --input_data_type. If you are trying to control the quantization/dequantization of real-numbers input arrays in the output file, use --inference_input_type.
2018-05-16 16:19:29.492371: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1270] Converting unsupported operation: Equal
2018-05-16 16:19:29.492813: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1270] Converting unsupported operation: LogicalAnd
2018-05-16 16:19:29.521856: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before Removing unused ops: 812 operators, 1241 arrays (0 quantized)
2018-05-16 16:19:29.550535: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After Removing unused ops pass 1: 802 operators, 1222 arrays (0 quantized)
2018-05-16 16:19:29.582482: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 802 operators, 1222 arrays (0 quantized)
2018-05-16 16:19:29.610492: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 1: 148 operators, 358 arrays (0 quantized)
2018-05-16 16:19:29.613697: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 2: 148 operators, 358 arrays (0 quantized)
2018-05-16 16:19:29.616876: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 3: 143 operators, 348 arrays (0 quantized)
2018-05-16 16:19:29.619981: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 4: 142 operators, 346 arrays (0 quantized)
2018-05-16 16:19:29.622935: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 5: 141 operators, 344 arrays (0 quantized)
2018-05-16 16:19:29.626348: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before dequantization graph transformations: 141 operators, 344 arrays (0 quantized)
2018-05-16 16:19:29.629622: I tensorflow/contrib/lite/toco/allocate_transient_arrays.cc:329] Total transient array allocated size: 3158144 bytes, theoretical optimal value: 3158144 bytes.
2018-05-16 16:19:29.631849: F tensorflow/contrib/lite/toco/tflite/export.cc:315] Some of the operators in the model are not supported by the standard TensorFlow Lite runtime. If you have a custom implementation for them you can disable this error with --allow_custom_ops. Here is a list of operators for which you will need custom implementations: ExpandDims, Slice, Stack, TensorFlowShape.
./convert-lite.sh: line 11: 70835 Abort trap: 6           /Volumes/Workspace/tensorflow/workspace/tensorflow/bazel-bin/tensorflow/contrib/lite/toco/toco --input_file=$(pwd)/model/frozen_inference_graph.pb --input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE --output_file=$(pwd)/model/deeplabv3_mnv2_pascal_trainval.tflite --inference_type=FLOAT --input_type=QUANTIZED_UINT8 --input_arrays=ImageTensor --output_arrays=SemanticPredictions --input_shapes=1,513,513,3 --default_ranges_min=0 --default_ranges_max=255

The model could not be loaded successfully. I think it is caused the warning: Here is a list of operators for which you will need custom implementations: ExpandDims, Slice, Stack, TensorFlowShape.

Is it possible to update node SemanticPredictions to use INT32 data type on Slice operation? Or do you have any suggestion on how to run it with TensorFlow lite?

[aquariusjay]

Hi dailystudio,

Thanks for bringing up this issue. Regarding your questions, I would have a few suggestions at this moment:

1. You could try to change the export_model.py to use int32 SemanticPredictions, and see what happens (e.g., there may be other issues).
2. You could contact TensorFlow Mobile and TensorFlow Lite to support those operations.

We will look into this issue more carefully in the near future.

Thanks,

[dailystudio]

Hello aquariusjay, Thanks for your valuable advices!

I modify the line in export_model.py (around line 131):

    semantic_predictions = tf.slice(
        tf.cast(predictions[common.OUTPUT_TYPE], tf.int32),
        [0, 0, 0],
        [1, resized_image_size[0], resized_image_size[1]])

add tf.cast to cast predictions to int32. And then export model with the following command:

python export_model.py --checkpoint_path model/model.ckpt-30000 --export_path model/frozen_inference_graph.pb

Here I only pointed the checkpoint path and export path, rest parameters kept with default value.

Now running the model with Tensorflow Mobile is successful!

The new issue is that the output array (SemanticPredictions) is a zero array, all the elements are 0. There is no any error or warning are printed during the inference.

Do you have any suggestion? Is it caused by some unsupported op in the model? Or should I add more parameters during exporting the model?

[aquariusjay]

You need to make sure you have provided the right flag values for the model variant (e.g., MobileNet-v2 or Xception_65) that you are using. Check local_test.sh or local_test_mobilenetv2.sh for reference.

[dailystudio]

Hello aquariusjay,

Now it works on my mobile devices. I add model variant parameter during export model and also fix a dimension issue of passing the width and heigh in wrong order during the inference.

Thanks for your help. I will accomplish it as complete demo.

[liangxiao05]

@dailystudio，hi,could you share which mobile do you run it on ,and how is the latency?

[dailystudio]

@liangxiao05, at Oneplus 3T and Oneplus 5, 900ms ~ 2700ms per inference.

[Shenghsin]

@dailystudio could you share your demo?

[JoseRNFaria]

@dailystudio I tried your solution and it worked. Thanks. In my pixel 2 the run time is ~1300 - 1500 ms if you reduce the input images sizes to 256x256 the time goes to ~400 - 500 ms.

I'm trying to pass the model to tflite but a error is making it impossible.

[dailystudio]

@JoseRNFaria Glad to here that my works can help you!

[Shenghsin]

@dailystudio I have the same problem "The new issue is that the output array (SemanticPredictions) is a zero array, all the elements are 0." Could you tell me how to fix it?? Thanks!

[dailystudio]

@Shenghsin @aquariusjay @JoseRNFaria , I have written a demo app of this model. I am still updating the document. Here is the repository link: https://github.com/dailystudio/ml/tree/master/deeplab

[llhe]

@dailystudio To run deeplabv3+ on mobile phones faster, you can checkout MACE which has deeplabv3+ in the MACE Model Zoo. Here are some benchmark results which includes deeplab-v3-plus-mobilenet-v2.

[austingg]

hope we can use tensorflowlite to run the deeplab v3+ model, since segmentation task ususally cost much more than classification task. quantization may help a lot.

[weehe91325]

@dailystudio, did you managed to use in your demo the model converted to TF Lite? Does anyone know if this is possible?

[sumsuddin]

Hi, @dailystudio. Just change this line to predictions[output] = tf.argmax(logits, 3, output_type=dtypes.int32).

[SanthoshRajendiran]

@dailystudio @aquariusjay @sumsuddin.. Is there support for TFLite conversion of DeeplabV3 with MobilenetV2? Could not find relevant documentation.

[melody-rain]

watching this

[hakS07]

@dailystudio

Hello aquariusjay, Now it works on my mobile devices. I add model variant parameter during export model and also fix a dimension issue of passing the width and heigh in wrong order during the inference. Thanks for your help. I will accomplish it as complete demo.

how do you solve the issue, because have tried to convert the pb file to tflite by using tf converter tflite_convert ----output_format=TFLITE --inference_type=FLOAT --inference_input_type=FlOAT --input_arrays=sub_2 --input_shapes=1,257,257,3 --output_arrays=ResizeBilinear_2 --output_file=mobilenet.tflite --graph_def=mobilenet.pb --mean_values=128 --std_dev_values=127 --allow_custom_ops --post_training_quantize and i got a slow inference on mobile(using iphone 6) can check this for me ??

[helloItzMe]

can anyone create a py script file for converting xception model frozen_graph.pb file to tflite??

[charlizesmith]

do you have any tflite file of the module