TFLite not support Dynamic input size

[WenguoLi]

System information

• TensorFlow version (you are using): 1.12
• Are you willing to contribute it (Yes/No): No

Describe the feature and the current behavior/state.

Try running tflite_convert again, with --input_shapes=1,none,none,3, but these conversion cannot be supported. For some networks requiring dynamic input, the static input of tflite is too limited, such as Superresolution Neural Network.

[jdduke]

Can you provide a link/reference to an existing (trained) model that requires dynamic shape elements? That will help us focus support for this feature. Thanks.

[jdduke]

To add some context, support for conversion with explicitly dynamic dimensions is on our radar.

[weiweizhou]

hello,I'm trying to deploy Superresolution on android, and I found it seems not support Dynamic input size, have you fix this problem now? @jdduke

[jdduke]

We're actively working to improve this. Expect a change in the next few days which will improve the ability to dynamically resize at runtime. You'll still need to give it a static shape during conversion, but the graph should be much more robust to dynamic size changes.

[gnsmrky]

@jdduke Any updates in supporting dynamic size at runtime?

[jdduke]

We've landed several fixes, but they've been rolled back due to downstream clients depending on the current (somewhat broken) behavior in our converter. We're hoping to resolve this properly in Q3, and might expose an experimental flag in the converter to generate graphs that always preserve rank/shape operators.

[Oktai15]

@jdduke any update?

[jdduke]

No concrete updates yet, but we are investing this quarter in improving the situation, and will be sure to address this specific case. Thanks for your patience.

Some workarounds that folks use are 1) for images, do the resize outside the graph and use a fixed shape input, 2) if you do require actual resizing, but only for a small number of fixed shapes, create a separate interpreter instance for each shape.

[yanmenglu]

@jdduke tflite support multi-size image input. Is there some sample about how to do that?

[yanmenglu]

@jdduke Is plan to support multi-size image input on tflite?

[renjithamadeus]

Is there any update on this? This feature would be very useful for supporting seq2seq models in TFLite.

[TerenceYang]

@jdduke any update?

[jdduke]

We're still actively working on this, and it remains a priority for Q1.

[gargn]

We added support for unknown dimensions in TensorFlow Lite today (55912083e2f16087c2f29394acf8a6a4811a2ce0).

Can you try converting your model again with tonight's (1/31) tf-nightly once it's released (pip install tf-nightly). Convert the model with experimental_new_converter = True.

When you load the model it should have an additional field shape_signature that contains the shape with any unknown dimensions marked with -1. shape will have those dimensions marked with 1.

You can then call ResizeInputTensor with the desired shape when running the interpreter. The generated model will only work on the latest TensorFlow version (i.e. the interpreter on the tf-nightly version you are running).

If it does not work, can you provide a detailed error and repro instructions?

[harsh306]

If I have model version tf 1.x, exported as saved_model format. Can I still use this functionality for dynamic input shapes?

[gargn]

@harsh306 Yes. As long as the 1.X TensorFlow SavedModel takes in dynamic input shapes, then it will work with TFLite after reconverting the model. I recommend using TFLiteConverter.from_saved-model in the tf-nightly (which supports 1.X SavedModels) to try converting the model and testing this functionality.

[harsh306]

Thanks, Yes, I will try that. Currently I am stuck on different issue of pip install tf-nightly on MAC :) Will update results once I figure out. https://github.com/tensorflow/tensorflow/issues/27470

[harsh306]

ok, I am using tf-nightly==2.2.0-dev20200204 Still getting the error during converting.

    converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
    converter.experimental_new_converter = True
    print(converter.experimental_new_converter) # outputs True
    converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
    tflite_quant_model = converter.convert()
    file = open(TFLITE_MODEL+'model.tflite', 'wb')
    file.write(tflite_quant_model)

Error:

File "/Users/hpathak/anaconda3/lib/python3.7/site-packages/tensorflow_core/lite/python/lite.py", line 478, in convert
    _get_tensor_name(tensor), shape_list))
ValueError: None is only supported in the 1st dimension. Tensor 'input_img' has invalid shape '[None, None, None, 3]'.

[gargn]

Unfortunately, dynamic shapes are currently not supported with quantization.

We are hoping to enable dynamic shapes with weight-only quantization support in the near future (which seems to be sufficient for your use case). However, full integer post training quantization support won't be added in the immediate future.

[dusanmacho]

@gargn For speech recognition applications, as opposed to image, it is natural using dynamic shapes due to varying lengths of the input signal. Would you point to an example that deals with this dynamic aspect and uses quantization and TFLite? I am aware of Command Recognition Android Demo, but the speech input is fixed to 1 second in that example. Thank you.

[dathudeptrai]

@dusanmacho how about if u set fixed size for length and it large enough (let say around 1minutes). Then u can apply masking input on the begining of the code ?

[dusanmacho]

Thanks! Unfortunately, I am unable to run quantization on a fixed shape input either. See details below.

TF inference model:

tf.__version__
'2.2.0-dev20200212'
inference_model = tf.keras.models.load_model('./inference_model.h5')
inference_model.summary()
Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
Input_feats (InputLayer)     [(None, None, 41, 1)]     0         
_________________________________________________________________
conv2d (Conv2D)              (None, None, 21, 32)      3872      
_________________________________________________________________
batch_normalization (BatchNo (None, None, 21, 32)      128       
_________________________________________________________________
activation (Activation)      (None, None, 21, 32)      0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, None, 11, 32)      78848     
_________________________________________________________________
batch_normalization_1 (Batch (None, None, 11, 32)      128       
_________________________________________________________________
activation_1 (Activation)    (None, None, 11, 32)      0         
_________________________________________________________________
reshape (Reshape)            (None, None, 352)         0         
_________________________________________________________________
lstm (LSTM)                  (None, None, 172)         361200    
_________________________________________________________________
lstm_1 (LSTM)                (None, None, 172)         237360    
_________________________________________________________________
lstm_2 (LSTM)                (None, None, 172)         237360    
_________________________________________________________________
dense (Dense)                (None, None, 29)          5017      
=================================================================
Total params: 923,913
Trainable params: 923,785
Non-trainable params: 128
_________________________________________________________________

The following runs correctly, and the TFLite model inference results match those of the original TF model (@gargn thumbs up!!!):

inference_model = tf.keras.models.load_model('./inference_model.h5')
converter = tf.lite.TFLiteConverter.from_keras_model(inference_model)
converter.experimental_new_converter = True
tflite_model = converter.convert()

The following gives an error due to dynamic shapes not being supported in TFLite quantization yet:

inference_model = tf.keras.models.load_model('./inference_model.h5')
converter = tf.lite.TFLiteConverter.from_keras_model(inference_model)
converter.experimental_new_converter = True
converter.experimental_new_quantizer = True
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()

*** ValueError: None is only supported in the 1st dimension. Tensor 'Input_feats' has invalid shape '[None, None, 41, 1]'.

Setting a fixed input shape gives a different error:

inference_model = tf.keras.models.load_model('./inference_model.h5')
inference_model.input.set_shape((1, 100, 41, 1))
converter = tf.lite.TFLiteConverter.from_keras_model(inference_model)
converter.experimental_new_converter = True
converter.experimental_new_quantizer = True
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()

... quantize_weights.cc:351] Quantize weights tool only supports tflite models with one subgraph.
Exception: Quantize weights transformation failed.

[dathudeptrai]

@dusanmacho be carefull with rnn, at this time, lstm haven’t fully support yet. Pls replace rnn to dilated convution (eg temporal convolution network).

[dusanmacho]

@gargn Is stateful LSTM supported with converter.experimental_new_converter? When using

tf.keras.layers.LSTM(..., return_state = False, stateful = False, ...) 

converter.convert() works. However when using

tf.keras.layers.LSTM(..., return_state = False, stateful = True, ...)

converter.convert() gives the following error:

ValueError: Input 0 of node model/lstm/AssignVariableOp was passed float from model/lstm/1991:0 incompatible with expected resource.

[abdullahshafin]

Hi everyone and @gargn, I am getting the same error in a slightly different use case as I am working with images:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gixhx/.conda/envs/tf22_p37/lib/python3.7/site-packages/tensorflow/lite/python/lite.py", line 655, in convert
    _get_tensor_name(tensor), shape_list))
ValueError: None is only supported in the 1st dimension. Tensor 'input_image' has invalid shape '[None, None, None, 3]'.

Tensorflow nightly version: 2.2.0-dev20200411

My code for conversion from SavedModel to tflite:

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_path)
print(converter.experimental_new_converter)  # outputs True
converter.optimizations = [tf.lite.Optimize.DEFAULT]
model_tflite = converter.convert()

If I convert slightly differently:

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_path)
print(converter.experimental_new_converter)  # outputs True
converter.optimizations = []  # this was the default behavior
model_tflite = converter.convert()

I get the error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gixhx/.conda/envs/tf22_p37/lib/python3.7/site-packages/tensorflow/lite/python/lite.py", line 704, in convert
    **converter_kwargs)
  File "/home/gixhx/.conda/envs/tf22_p37/lib/python3.7/site-packages/tensorflow/lite/python/convert.py", line 536, in toco_convert_impl
    enable_mlir_converter=enable_mlir_converter)
  File "/home/gixhx/.conda/envs/tf22_p37/lib/python3.7/site-packages/tensorflow/lite/python/convert.py", line 241, in toco_convert_protos
    raise ConverterError("See console for info.\n%s\n%s\n" % (stdout, stderr))
tensorflow.lite.python.convert.ConverterError: See console for info.
2020-04-11 14:59:11.353946: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-04-11 14:59:11.353969: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2020-04-11 14:59:12.413053: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:307] Ignored output_format.
2020-04-11 14:59:12.413085: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:310] Ignored drop_control_dependency.
loc("resize_image_with_pad/control_dependency"): error: requires all operands to be either same as or ref type of results
Traceback (most recent call last):
  File "/home/gixhx/.conda/envs/tf22_p37/bin/toco_from_protos", line 8, in <module>
    sys.exit(main())
  File "/home/gixhx/.conda/envs/tf22_p37/lib/python3.7/site-packages/tensorflow/lite/toco/python/toco_from_protos.py", line 93, in main
    app.run(main=execute, argv=[sys.argv[0]] + unparsed)
  File "/home/gixhx/.conda/envs/tf22_p37/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/gixhx/.conda/envs/tf22_p37/lib/python3.7/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/gixhx/.conda/envs/tf22_p37/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/home/gixhx/.conda/envs/tf22_p37/lib/python3.7/site-packages/tensorflow/lite/toco/python/toco_from_protos.py", line 56, in execute
    enable_mlir_converter)
Exception: <unknown>:0: error: loc("resize_image_with_pad/control_dependency"): requires all operands to be either same as or ref type of results
<unknown>:0: note: loc("resize_image_with_pad/control_dependency"): see current operation: %1 = "tf.Identity"(%arg0) {device = ""} : (tensor<?x?x?x3x!tf.quint8>) -> tensor<?x?x?x3xui8>

More Information: I have a classification usecase with ResNet50 model which I trained with transfer learning using tf.keras (tensorflow 2.x). Input image dimensions are 224x224x3. Input data is images and resized to 224x224x3 with the following function that maintains original image aspect ratio: image = tf.image.resize_with_pad(image, target_height=224, target_width=224) This means, the model is trained on images with black padding on shorter dimension. Therefore, I need to give similar images at inference time, otherwise the model accuracy drops significantly. Therefore, I converted the tf.keras .h5 model into SavedModel format with the following serving function:

def serving_input_receiver_fn():
    """Preprocess/resize images (like during tfrecord creation) and prepare for inference"""
    input_image = tf.compat.v1.placeholder(dtype=tf.uint8,
                                           shape=[None, None, None, 3],
                                           name='input_image')

    image = tf.image.resize_with_pad(image=input_image,
                                     target_height=224,
                                     target_width=224,
                                     method=tf.image.ResizeMethod.LANCZOS3,
                                     antialias=True)
    image = image/255.0

    return tf.estimator.export.ServingInputReceiver({'input_1': image},
                                                    {'image_original': input_image})

Code to convert from h5 model to SavedModel model:

# code to convert from tf.keras h5 model to tf SaveModel .pb model
estimator = tf.keras.estimator.model_to_estimator(keras_model=model_h5,
                                                  model_dir=model_rootpath)
exported_path = estimator.export_saved_model(sm_model_fp, serving_input_receiver_fn)

Now, this model performs as expected if I use the SavedModel for inference directly on my ubuntu cloud instance. The issue only arises when I try to convert this model from SavedModel to tflite model for deploying on iOS and Android, as I have shown on the top of my comment.

As you can see, my serving function takes randomly sized input images becauses it resizes them to required size while maintaining the aspect ration. Therefore, I need to have [None, None, None, 3] for input dimensions.

Can anyone share if there's a way to do this and what I might be missing? Is there really this functionality missing at this point? I thought this would be a very common use case (it does work flawlessly using SavedModel).

[purva98]

Hey @gargn @karimnosseir @miaout17 @dusanmacho @Saduf2019 does it support dynamic input shape, while using frozen graphs for conversion to tflite in TF 1.x version. ( i have a .ckpt model built in TF 1.14 , I converted it to a .pb model, in order to convert it to .tflite model) Is this how I should be converting a dynamic input shape .ckpt model to a tflite model?

[ethanyanjiali]

Unfortunately, dynamic shapes are currently not supported with quantization.

We are hoping to enable dynamic shapes with weight-only quantization support in the near future (which seems to be sufficient for your use case). However, full integer post training quantization support won't be added in the immediate future.

i think this issue might be related: https://github.com/tensorflow/tensorflow/issues/40268 try use float32 instead of uint8 for input

[alfarok]

I am running into a similar issue where input sizes are updating to accomodate for a dynamic batch size, however the output batch dimension appears to remain fixed at 1 despite the batch size.

(1, 512, 512, 3) as opposed to (X, 512, 512, 3).

I outlined all the details in another similar issue here.

[karimnosseir]

@alfarok updated the other issue. "You should have your model converted again with supporting dynamic batch size. Looks like you specified static size during conversion."

Thanks

[yuqiu1233]

hi @gargn,
I find that dynamic input shape doesn't suppot GPU accelerate. Any updates in supporting GPU or NNAPI accelerate when using dynamic size input?

[JuliRao]

It works, helps a lot!!

[karimnosseir]

@yuqiu1233 GPU and NNAPU delegates doesn't support handling dynamic batch size currently. Depending on your model, you might want to consider splitting your model so you can get the heavy part accelerated and post processing which requires dynamic batch on CPU.

[dathudeptrai]

@yuqiu1233 GPU and NNAPU delegates doesn't support handling dynamic batch size currently. Depending on your model, you might want to consider splitting your model so you can get the heavy part accelerated and post processing which requires dynamic batch on CPU.

does GPU and NNAPI delegates support handling dynamic input shape such as [1, None, 100] ?.