Feature Request - Deeplab TFLITE Android Application

[SanthoshRajendiran]

Feature request, requesting an android application for Deeplab tag:feature_template

System information

• TensorFlow version (you are using): Tensorflow 1.12 [tensorflow-lite-1.12 ]
• Model Information: Deeplab V3 MobilenetV2
• Are you willing to contribute it (Yes/No): No
• Android API Version: API 24 (Android Nougat)

Feature Current Behavior There is no mobile application to test out the working of Deeplab tflite model in Android or IOS. This seems as a direct need for developers and it will be helpful for knowing the parsing mechanism for tflite where we get semantic predictions as an output, as there is an unclear way of parsing the specific data type in android and ios(prediction?) as it involves pixel data.

Current API: Need of change We are in need of new API model to help in with parsing the model input. Could be released as a subsequent fix.

Beneficiaries People who are developing camera applications can directly benefit from this as it involves playing with segmentation on android devices.

Other information. We are trying building the application tweaking with available applications. We are currently facing up some issues. Hereby with, we are attaching the issue links.

https://stackoverflow.com/questions/53228969/unable-to-test-and-deploy-a-deeplabv3-mobilenetv2-tensorflow-lite-segmentation-m

https://stackoverflow.com/questions/53236290/unable-to-load-tflite-deeplab-segmentation-model-in-android-application-error

[shashishekhar]

• @jdduke

@SanthoshRajendiran : We have demo apps: One example: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/java/demo The 2 issues that you reference seem to be due to model conversion. Can you attach your TF lite model with this issue

[jdduke]

I believe @suharshs has looked into Deeplab support with TensorFlow Lite.

[SanthoshRajendiran]

Thanks for the prompt response @shashishekhar @jdduke. I have gone through the app link you have provided. That is specific for Image Classification, I think it cannot be directly used for image segmentation.

With response to your query, I am attaching here both the frozen graph (From the official repo - both 8MB and 3MB) and the tflite we created from the 8MB model. We have not been able to convert the 3MB Model. Will be helpful if you could support us with the conversion command for 3MB model.

The command for conversion to tflite, is given in the stackoverflow link provided above

deeplab_tflite_issues.zip

[SanthoshRajendiran]

Hello guys. Waiting for your reply.. Any improvements as of now. Do you need any help from our side??

[kismeter]

I have the same issue with you on model convert. for the real time segmentation you can ref to https://github.com/tantara/JejuNet, seems that project has successfully converted the model to tflite. but the accuracy is not that good.

[SanthoshRajendiran]

Hi @kismeter . Ya. I happened to see the code in Jejunet, but could not find it useful with relevance to the provided deeplab model. It seems that the model is tweaked in various stages to make it run on the device. Anyway, in Jejunet, only tflite is available, no references to Deeplab or pb file. I am in need of converting pb file to tflite and deploying it on the device.

[normandra]

I am also facing the same issue. Specifically I'm trying to convert

http://download.tensorflow.org/models/deeplabv3_mnv2_cityscapes_train_2018_02_05.tar.gz

into a tflite file. Optimally also quantization but before that I already ran onto this problem

using this:

tflite_convert --output_file model/test2.tflite
 --graph_def_file deeplabv3_mnv2_cityscapes_train_2018_02_05/deeplabv3_mnv2_cityscapes_train/frozen_inference_graph.pb  

--input_arrays ImageTensor 

--output_arrays SemanticPredictions 

--input_shapes=1,513,513,3

I get

Check failed: array.data_type == array.final_data_type Array "ImageTensor" has mis-matching actual and final data types (data_type=Uint8, final_data_type=Float).

any help would be highly appreciated.

[SanthoshRajendiran]

Hi @normandra We are going through the same process. Actually, we have been able to overcome the same issue you are facing, by providing the parameters: inference-input type and inference type. For the conversion command, do check out the stack overflow link mentioned above. Actually, we are facing some other issue in tflite conversion as discussed above and are currently waiting for @suharshs to reply back.

In your case, if you have not explicitly mentioned the inference type parameter. by default it will take the input type as inference input type. By default, FLOAT becomes the inference type (You can set it to QUANTIZED UINT8 also). Actually, updates in tflite conversion. forces on usage of Quantized UINT8 and Float data types.

[SanthoshRajendiran]

@tensorflowbutler @suharshs Any updates on the feature request???

[melody-rain]

watching this...

[normandra]

Thanks for the info @SanthoshRajendiran . I already tried specifying the input / inference type and I seem to be getting either the same error or the Nonetype error.

EDIT: Okay now I'm facing the exact same problem you are ( ByteBuffer is not a valid flatbuffer model ). I guess we can only wait now.

[suharshs]

Hi sorry for the delay, thanksgiving holidays and such. We will take a look into reproducing your issue. Thanks!

[melody-rain]

@sandeepngupta @normandra Hi I found it is easy to solve the problem. In my case tf.image.ResizeMethod.NEAREST_NEIGHBOR and the slice operator are not supported, therefore when exporting the model with export_model.py, I do not include the two operators. Below is how I modify the code to export proper '.pb'.

    # Crop the valid regions from the predictions.
    # enable tflite for exporting tflite model
    if not FLAGS.tflite:
        semantic_predictions = tf.slice(
            predictions[common.OUTPUT_TYPE],
            [0, 0, 0],
            [1, resized_image_size[0], resized_image_size[1]])

    # Resize back the prediction to the original image size.
    def _resize_label(label, label_size):
      # Expand dimension of label to [1, height, width, 1] for resize operation.
      label = tf.expand_dims(label, 3)
      resized_label = tf.image.resize_images(
          label,
          label_size,
          method=tf.image.ResizeMethod.NEAREST_NEIGHBOR,
          align_corners=True)
      return tf.squeeze(resized_label, 3)
    if FLAGS.tflite:
        semantic_predictions = tf.cast(predictions[common.OUTPUT_TYPE], tf.uint8)
        semantic_predictions = tf.squeeze(semantic_predictions)
    else:
        semantic_predictions = _resize_label(semantic_predictions, image_size)

    semantic_predictions = tf.identity(semantic_predictions, name=_OUTPUT_NAME)

The operators not supported can be implemented in your Java or Python code.

[normandra]

Thanks for sharing @melody-rain ,

do you mind on elaborating how to implement the unsupported ops in java / python ? I'm hit with a segenv in my android wrapper so I'm not sure what to do there. Testing in python for some weird reason regardless of what my input tensor is set to i get the same output.

EDIT: I guess not exactly the same everytime but very similiar

[melody-rain]

@normandra slice op is used to crop the image, so you can use python's slice. resize can also be replace with opencv's resize.

[normandra]

Okay but at what point would I implement those? My output from the model currently is a 513x513 tensor pictured above.

[alanchiao]

Seems like there are number of subproblems in this issue. I'm trying to convert the 3MB mobilenetv2_dm05_coco_voc_trainval model and see what may be missing

[alanchiao]

For the 3MB model, I found the cause of the TOCO error. I'm working with internal engineers to figure out what the proper fix is.

[SirNeuman]

I am also running into issues with converting the frozen graphs, although they're a bit different depending on which graph i try to convert to tensorflow lite. When converting the graph found from http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_train_aug_2018_01_29.tar.gz via TOCO with the command:

tflite_convert \
  --output_file=test.lite \
  --graph_def_file=frozen_inference_graph.pb \
  --input_arrays=ImageTensor \
  --output_arrays=SemanticPredictions \
  --input_shapes=1,513,513,3 \
  --inference_input_type=QUANTIZED_UINT8 \
  --inference_type=FLOAT \
  --mean_values=128 \
  --std_dev_values=128

I get no issues in the conversion. But when i try and implement it in my app on Android I get the java.lang.IllegalArgumentException: Internal error: Failed to run on the given Interpreter: tensorflow/contrib/lite/kernels/depthwise_conv.cc:99 params->depth_multiplier * SizeOfDimension(input, 3) != SizeOfDimension(filter, 3) (0 != 32)Node number 30 (DEPTHWISE_CONV_2D) failed to prepare. error.

However when i try to retrain the model on my local machine with tensorflow 1.12 using the local_test_mobilenetv2.sh script in the tensorflow-models repo for Deeplab i get the following error when trying to convert the exported graph with the same tflite_convert command:

2018-12-07 10:29:51.195230: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Traceback (most recent call last):
  File "/usr/local/bin/tflite_convert", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/tensorflow/contrib/lite/python/tflite_convert.py", line 412, in main
    app.run(main=run_main, argv=sys.argv[:1])
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/usr/local/lib/python2.7/site-packages/tensorflow/contrib/lite/python/tflite_convert.py", line 408, in run_main
    _convert_model(tflite_flags)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/contrib/lite/python/tflite_convert.py", line 162, in _convert_model
    output_data = converter.convert()
  File "/usr/local/lib/python2.7/site-packages/tensorflow/contrib/lite/python/lite.py", line 453, in convert
    **converter_kwargs)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/contrib/lite/python/convert.py", line 342, in toco_convert_impl
    input_data.SerializeToString())
  File "/usr/local/lib/python2.7/site-packages/tensorflow/contrib/lite/python/convert.py", line 135, in toco_convert_protos
    (stdout, stderr))
RuntimeError: TOCO failed see console for info.
2018-12-07 10:29:53.090897: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: ResizeNearestNeighbor
2018-12-07 10:29:53.104921: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before Removing unused ops: 811 operators, 1236 arrays (0 quantized)
2018-12-07 10:29:53.131609: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After Removing unused ops pass 1: 801 operators, 1217 arrays (0 quantized)
2018-12-07 10:29:53.160096: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 801 operators, 1217 arrays (0 quantized)
2018-12-07 10:29:53.186542: F tensorflow/contrib/lite/toco/graph_transformations/propagate_fixed_sizes.cc:624] Check failed: input_shape.dims().size() == op->size.size() (4 vs. 3)

None

This is the same error i get when i try to run the tflite_convert command on the frozen graph from http://download.tensorflow.org/models/deeplabv3_mnv2_dm05_pascal_trainaug_2018_10_01.tar.gz.

Also just a side note: I tried running the tflite_convert command on the frozen graph from http://download.tensorflow.org/models/deeplabv3_pascal_train_aug_2018_01_04.tar.gz just to see what would happen, and the conversion seemed to work. But i got to the same error point when actually running inference in the app with the model with nearly the same error message java.lang.IllegalArgumentException: Internal error: Failed to run on the given Interpreter: tensorflow/contrib/lite/kernels/depthwise_conv.cc:99 params->depth_multiplier * SizeOfDimension(input, 3) != SizeOfDimension(filter, 3) (0 != 64)Node number 33 (DEPTHWISE_CONV_2D) failed to prepare. Just a difference of SizeOfDimension(filter, 3) (0 != 64) vs SizeOfDimension(filter, 3) (0 != 32) in the mobilenetsv2 model.

I'm relatively new to TensorFlow/ML so forgive me for any extraneous info or misuse of terminology. I'm just trying to start out with trying to implement the Deeplabv3+ model on Android with a pretrained model for now. Not sure if there is a workaround? I looked at https://github.com/dailystudio/ml/tree/master/deeplab implementation of the Deeplab model on Android, but it appears to use the soon to be deprecated version Tensorflow Mobile API as opposed to Tensorflow Lite. I also am unsure as to whether a model that is trained with the current version of Tensorflow would work with this demo app (I'll soon try) since this codebase is from May (not sure which version was used to generate the graph that was used in the demo).

[SirNeuman]

So I did follow @melody-rain's advice and added their code into export_model.py and ran the tflite_convert command on the frozen graph. Had to make some other adjustments to my java code and got the model to successfully run in my android app without crashing/raising errors. The resulting graph for the tflite conversion appears to produce something... i'm just not sure what.

here is an inference result from the frozen graph without the lite conversion, which i ran via python on my Macbook Pro:

and here is the same inference result from the graph with the conversion, which i ran through my android app on a Samsung Galaxy Tab A:

Its an improvement from not working at all, however i'm not sure what exactly the results from the lite conversion means. The results from the non converted model are pretty decent. Not sure if my conversion from the output results in the java code is working incorrectly or if it is just an issue with the conversion.

[jdduke]

@SirNeuman: Just to confirm, have you tried without enabling quantization during conversion (i.e., using the float path)?

[SirNeuman]

Sorry. I was actually messing up my input image when i resized it for running through the inference on mobile. It actually works quite well now. This is a different input image running for inference, but you can see that the results actually make a bit more sense and is close to what i'm trying to achieve:

I still need to implement the java code for scaling the output image...

@jdduke: I assume you mean changing --inference_input_type=QUANTIZED_UINT8 to --inference_input_type=FLOAT in my conversion command? I just went ahead and tried it to see what the results would be like. I had to change the input_image when exporting the model from type uint8 to float32 to get the conversion to run. I then also had to update my convertBitmapToByteBuffer method when running the image through the model in the app. I ended up getting it to run without errors and ended up with pretty similar results (not sure if it's more accurate but it definitely runs inference faster?). Just to note this produces an 8.5mb lite graph as opposed to a 2.2mb lite graph that my original conversion produced:

Here's with the unconverted (non-lite) model for reference:

Also the time for inference with the original uint8 type was 10335 ms, while the time for inference with the float type was 4455ms on my Galaxy Samsung Tab A. Which does seem strange to me that the quantized version performs worse, but once again I'm relatively new to this so perhaps i'm either doing something wrong or i'm misunderstanding the changes i'm making.

[Srinivas-Introtuce]

For the 3MB model, I found the cause of the TOCO error. I'm working with internal engineers to figure out what the proper fix is.

Hello @alanchiao , we are keen to know the response from your internal engineers team for the TOCO error. How long do you think you need to provide a solution to successfully convert 3MB pb model into tflite? Thanks for your continued support.

[Srinivas-Introtuce]

Sorry. I was actually messing up my input image when i resized it for running through the inference on mobile. It actually works quite well now. This is a different input image running for inference, but you can see that the results actually make a bit more sense and is close to what i'm trying to achieve:

I still need to implement the java code for scaling the output image...

@jdduke: I assume you mean changing --inference_input_type=QUANTIZED_UINT8 to --inference_input_type=FLOAT in my conversion command? I just went ahead and tried it to see what the results would be like. I had to change the input_image when exporting the model from type uint8 to float32 to get the conversion to run. I then also had to update my convertBitmapToByteBuffer method when running the image through the model in the app. I ended up getting it to run without errors and ended up with pretty similar results (not sure if it's more accurate but it definitely runs inference faster?). Just to note this produces an 8.5mb lite graph as opposed to a 2.2mb lite graph that my original conversion produced:

Here's with the unconverted (non-lite) model for reference:

Also the time for inference with the original uint8 type was 10335 ms, while the time for inference with the float type was 4455ms on my Galaxy Samsung Tab A. Which does seem strange to me that the quantized version performs worse, but once again I'm relatively new to this so perhaps i'm either doing something wrong or i'm misunderstanding the changes i'm making.

@SirNeuman Did you include any custom implementations for Slice and Resize operators which were excluded in export_model.py file? Can we please know the Samsung Tab A device specifications, as we are equally surprised to know the long inference times?

[alanchiao]

@Srinivas-Introtuce: since the day I provided the last response, I've been trying to verify the below analysis and fix with a SWE from another team and unfortunately they haven't gotten back. Provided it is correct, it would expect it to take at most two days after, including a day to a day and a half for code review.

If you're not familiar with or interested in TOCO internals, please ignore the following:

In a build of the latest source code, I saw the following error: Check failed: input_shape.dims().size() == op->size.size() (4 vs. 3) at propagate_fixed_sizes.cc:625.

propagate_fixed_sizes is a graph transformation that takes the shapes of the model input tensors and propagates them throughout the graph to compute the shapes of each op's input and output tensors.

The error is thrown for the Slice Op that follows an Argmax Op. For this graph, the argmax input shape is [1, 513, 513, 21] and the computed output shape from propate_fixed_sizes.cc is [1, 513, 513, 1]. This input shape goes to the slice op (input_shape.dims().size() = 4 with the 4 elements), with the existing size param = [1, 256, 256] (op->size.size = 3 with the 3 elements), leading to the 4 != 3 error.

Typically, for argmax, the output shape would be [1, 513, 513], but I'm guessing that the 1 was appended to the end of the output shape since TOCO's in-memory representation works with 4D tensors. The fix would to be to modify the Slice operator's in-memory representation to match the Argmax behavior.

[Srinivas-Introtuce]

The error is thrown for the Slice Op that follows an Argmax Op. For this graph, the argmax input shape is [1, 513, 513, 21] and the computed output shape from propate_fixed_sizes.cc is [1, 513, 513, 1]. This input shape goes to the slice op (input_shape.dims().size() = 4 with the 4 elements), with the existing size param = [1, 256, 256] (op->size.size = 3 with the 3 elements), leading to the 4 != 3 error.

Hi @alanchiao - Thank you very much for the response. Yes myself and my team got fair understanding of the error based on your detailed description. We are looking forward to receive the respective fix and waiting to successfully convert 3mb pb file into tflite. Again, thank you for the continued support.

[SirNeuman]

@Srinivas-Introtuce yeah. i just cropped and resized the image before/after running the image through the graph in our android application code.

My GalaxyTab A specs are: Model Sm-T580 1.6GHz Octa Core Processor 2GB RAM Android 8.1

[normandra]

Finally after fiddling around I was able to make my implementation work as well.

Interestingly enough my result is also similar to @SirNeuman 's where the float model performed faster than the quantized one.

On a HTC U11 (Snapdragon 835) https://www.gsmarena.com/htc_u11-8630.php

with the input_size set to 300x300 I was able to reach: ~470ms on the float model (8.5 mb) ~830ms on the Quantized model (2.2 mb)

[jdduke]

If you don't mind sharing your converted .tflite model (either here or sending to me directly), we'd be happy to dive into performance issues.

Otherwise, it might be helpful if you could profile the model per these instructions (under the "Profiling model operators" section). Thanks!

[alanchiao]

@normandra , @Srinivas-Introtuce : to make sure, when you say quantized model, you are referring to models generated using the post_training_quantize flag right?

For performance, are you measuring with single-threaded performance? It makes a difference between with post_training_quantize, we currently don't support multithreading whereas we do for the float model. This is because in practice on a user's phone with various other applications running, multithreading is often slower than single threaded performance due to the contention.

[normandra]

Yes, I will upload my converted tflite tomorrow

On Dec 13, 2018 7:27 PM, alanchiao notifications@github.com wrote:

@normandrahttps://github.com/normandra , @Srinivas-Introtucehttps://github.com/Srinivas-Introtuce : to make sure, when you say quantized model, you are referring to models generated using the post_training_quantize flag right?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/tensorflow/tensorflow/issues/23747#issuecomment-447069814, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AH5PtKHVKa2e-dreNqYBt8IDOwFCkUsCks5u4pwMgaJpZM4YeDVJ.

[normandra]

So here are the two models:

models.zip

So I used the tip from @melody-rain to remove the unsupported ops and to convert the models.

To convert the model I used:

tflite_convert --output_file 300_quantized.tflite --graph_def_file 300.pb --input_arrays ImageTensor --output_arrays SemanticPredictions --input_shapes=1,300,300,3 --inference_input_type=QUANTIZED_UINT8 --inference_type=FLOAT --mean_values=128 --std_dev_values=128 --post_training_quantize

and for the other without the --post_training_quantize flag

[anilsathyan7]

Hi @normandra ,

We tried running your tflite model on OnePlus 3 (Snapdragon 820) and the inference time was around 2000 ms (both for single and double threaded execution) for 2.2 Mb model. How did you measure the execution time (is it just the running time for 'tflite.run()' of tflite intrepreter)?Also did you use nnapi (HTC U11 seems to support Android Pie) and how many threads were used in your case?

(So far the best performance that we found in terms of speed, is around 150 ms using Jejunet with input size of 256*256 and and INT64 output )

[normandra]

@anilsathyan7 I used 4 Threads. NNAPI doesn't seem to make any difference whatsoever on this device. To test the runtime I have made an app similiar to the one used in Jejunet which again is similiar to the one from the tflite / tfmobile demo provided by the tensorflow team.

Something to note about Jejunet is that its a modified version of deeplab. Some layers are missing there (Batchnormalization for example) presumably to boost the speed. It also seem that the model was trained from scratch.

[normandra]

@alanchiao I did test the quantized model on a single threaded run, and the quantized model did not perform better there.

[SanthoshRajendiran]

Hello @alanchiao . Any updates on the feature request?? And thanks @normandra for the information.

[Jayanth-L]

@melody-rain I followed your code and modified the export_model.py file accordingly

but got an error

INFO:tensorflow:Exported model performs single-scale inference.
WARNING:tensorflow:From /home/jayanthl/Documents/ai/tensorflow_models/models/research/deeplab/core/feature_extractor.py:160: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/jayanthl/.pythonvenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
Traceback (most recent call last):
  File "/home/jayanthl/Documents/ai/tensorflow_models/models/research/deeplab/export_model.py", line 166, in <module>
    tf.app.run()
  File "/home/jayanthl/.pythonvenv/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/home/jayanthl/Documents/ai/tensorflow_models/models/research/deeplab/export_model.py", line 142, in main
    semantic_predictions = tf.cast(predictions[common.OUTPUT_TYPE], tf.uint8)
  File "/home/jayanthl/.pythonvenv/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py", line 618, in _slice_helper
    _check_index(s)
  File "/home/jayanthl/.pythonvenv/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py", line 516, in _check_index
    raise TypeError(_SLICE_TYPE_ERROR + ", got {!r}".format(idx))
TypeError: Only integers, slices (`:`), ellipsis (`...`), tf.newaxis (`None`) and scalar tf.int32/tf.int64 tensors are valid indices, got 'semantic'```

[Jayanth-L]

@SirNeuman Can you please post the tflite_convert command which you used to convert the model ?

[Jayanth-L]

@melody-rain if i use python 3.6.8 i get the following error

INFO:tensorflow:Exported model performs single-scale inference.
Traceback (most recent call last):
  File "/home/jayanthl/Documents/ai/tensorflow_models/models/research/deeplab/export_model.py", line 166, in <module>
    tf.app.run()
  File "/home/jayanthl/.source_pythonvenv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/home/jayanthl/Documents/ai/tensorflow_models/models/research/deeplab/export_model.py", line 142, in main
    semantic_predictions = tf.cast(predictions[common.OUTPUT_TYPE], tf.uint8)
  File "/home/jayanthl/.source_pythonvenv/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 491, in _slice_helper
    end.append(s + 1)
TypeError: must be str, not int

@SirNeuman can you please post the export_model.py file ?

Update It was my fault, figured it out.

[melody-rain]

@Jayanth-L

Change

    predictions = tf.cast(predictions[common.OUTPUT_TYPE], tf.float32)

to:

    if not FLAGS.tflite:
        semantic_predictions = tf.slice(
            predictions[common.OUTPUT_TYPE],
            [0, 0, 0],
            [1, resized_image_size[0], resized_image_size[1]])

[alanchiao]

@SanthoshRajendiran : starting to code the fix now. Apologize for the delay

[haikuoyao]

Hi guys, you guys might need to see this post. https://groups.google.com/a/tensorflow.org/forum/?utm_medium=email&utm_source=footer#!msg/discuss/rzLivfAUGLk/EOZN2LyWCgAJ

One guy release tf-lite model of segmentation

[normandra]

I took a look at it. Aside from the final Reshape layer that model is pretty much identical with the one from JeJunet. So I guess to speed up the model we probably have to remove the BatchToSpaceNd + Mul + Add blocks in the feature extraction part. Some advice to do so would be great.

[alanchiao]

@Srinivas-Introtuce @SanthoshRajendiran : after this commit, you'll be able to convert the 3MB Deeplab TF graphdef to a TFLite model. I unfortunately don't have the cycles right now to hook everything up into the demo app.

[anilsathyan7]

We tried the official tensorflow experimental gpu backend for inference on andorid by following the following links:-

1. https://medium.com/tensorflow/tensorflow-lite-now-faster-with-mobile-gpus-developer-preview-e15797e6dee7
2. https://www.tensorflow.org/lite/performance/gpu

The speed up for mobilenet v1 and mobilenet v2 was around '2x' when compared to float model on OnePlus3(Snapdrahon 820, Adreno 530).Here are the benchmarks for mobilenet v2:-

Quantized CPU: 80 ms Float CPU: 80ms Float GPU:40ms

However when we tried the given deeplab segmentation model, the time taken for a frame was around 500ms in CPU float and 400ms in GPU on this device.This is still less than jejunet quantized inference(150ms).

We hope these are some other techniques for improving the performance 1.Training using a single class.(application specific) 2.Converting output from 125625621 to 165536 (jejunet style) 3.Removing some layers or operators (as per previous comments)

Our target is 30 FPS on android.Are there any other ways the improve the performance?

[ilous12]

@anilsathyan7 Did you implement android code with deeplabv3_257_mv_gpu.tflite? I did try but time taken 900ms.

I had a problem to inference. Can you see my code and give me comment?

Please reference below.

https://github.com/tensorflow/tensorflow/issues/25193

thanks.

[VolodymyrPavliukevych]

Added iOS example, waiting for pull request 25785 approve.

[elmeriz]

I am also running into issues with converting the frozen graphs, although they're a bit different depending on which graph i try to convert to tensorflow lite. When converting the graph found from http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_train_aug_2018_01_29.tar.gz via TOCO with the command:

tflite_convert \
  --output_file=test.lite \
  --graph_def_file=frozen_inference_graph.pb \
  --input_arrays=ImageTensor \
  --output_arrays=SemanticPredictions \
  --input_shapes=1,513,513,3 \
  --inference_input_type=QUANTIZED_UINT8 \
  --inference_type=FLOAT \
  --mean_values=128 \
  --std_dev_values=128

I get no issues in the conversion. But when i try and implement it in my app on Android I get the java.lang.IllegalArgumentException: Internal error: Failed to run on the given Interpreter: tensorflow/contrib/lite/kernels/depthwise_conv.cc:99 params->depth_multiplier * SizeOfDimension(input, 3) != SizeOfDimension(filter, 3) (0 != 32)Node number 30 (DEPTHWISE_CONV_2D) failed to prepare. error.

How did you resolve this issue @SirNeuman ? I am having the same problem trying to inference Deeplab v3+ ResNet 101 variant in tflite format. Conversion is succesful with no errors and the model works fine before conversion. I tried to adjust the export_model file according to melody-rain's tips, but it did not fix the problem.

[SanthoshRajendiran]

Could someone provide the command to convert deeplab pb file into a fully quantized tflite, along with some insights on the mean, std_dev_values and min-max ranges.

[normandra]

for quantized runtime I believe you also need to do quantization aware training, see https://github.com/tensorflow/tensorflow/issues/20867#issue-341770125