Error in TFLite of Fastspeech2 model

[ZhaoZeqing]

I used the code here , saved the .h5 model to tflite model then inference, but this error occurred:

Traceback (most recent call last): File "inference.py", line 56, in decoder_output_tflite, mel_output_tflite = infer(input_text) File "inference.py", line 40, in infer interpreter.allocate_tensors() File "anaconda3/lib/python3.7/site-packages/tensorflow/lite/python/interpreter.py", line 231, in allocate_tensors return self._interpreter.AllocateTensors() RuntimeError: tensorflow/lite/kernels/expand_dims.cc:42 axis <= input_dims.size was not true.Node number 0 (EXPAND_DIMS) failed to prepare.

[dathudeptrai]

@ZhaoZeqing did you run exactly our colab notebook ?, i tested it yesterday and it ran fine.

[ZhaoZeqing]

This is my code:

import numpy as np import tensorflow as tf import os

os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID" os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

from tensorflow_tts.processor import DB4Processor

def load_model(tflite_model_path): interpreter = tf.lite.Interpreter(model_path=tflite_model_path)

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

return interpreter, input_details, output_details

def prepare_input(input_ids): input_ids = tf.expand_dims(tf.convert_to_tensor(input_ids, dtype=tf.int32), 0) return (input_ids, tf.convert_to_tensor([0], tf.int32), tf.convert_to_tensor([1.0], dtype=tf.float32), tf.convert_to_tensor([1.0], dtype=tf.float32), tf.convert_to_tensor([1.0], dtype=tf.float32))

def inference(input_text, interpreter, input_details, output_details): for x in input_details: print(x) for x in output_details: print(x) processor = DB4Processor(None, '') input_ids = processor.text_to_sequence(input_text) interpreter.resize_tensor_input(input_details[0]['index'], [1, len(input_ids)]) interpreter.resize_tensor_input(input_details[1]['index'], [1]) interpreter.resize_tensor_input(input_details[2]['index'], [1]) interpreter.resize_tensor_input(input_details[3]['index'], [1]) interpreter.resize_tensor_input(input_details[4]['index'], [1]) interpreter.allocate_tensors() input_data = prepare_input(input_ids) for i, detail in enumerate(input_details): interpreter.set_tensor(detail['index'], input_data[i])

interpreter.invoke()
return interpreter.get_tensor(output_details[0]['index']), interpreter.get_tensor(output_details[1]['index'])

def synthesis(input_text, interpreter, input_details, output_details): decoder_output_tflite, mel_output_tflite = inference(input_text, interpreter, input_details, output_details) mels = tf.reshape(decoder_output_tflite, [-1, 80]).numpy() print(mels.shape)

if name == "main": tflite_model_path = 'tflite_models/fastspeech2_19w_quant.tflite' interpreter, input_details, output_details = load_model(tflite_model_path) synthesis(text, interpreter, input_details, output_details)

All of the output is here:

2020-07-22 11:27:51.913437: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 anaconda3/lib/python3.7/site-packages/tensorflow_addons/utils/ensure_tf_install.py:44: UserWarning: You are currently using a nightly version of TensorFlow (2.3.0-dev20200523). TensorFlow Addons offers no support for the nightly versions of TensorFlow. Some things might work, some other might not. If you encounter a bug, do not file an issue on GitHub. UserWarning, {'name': 'input_ids', 'index': 0, 'shape': array([1, 1], dtype=int32), 'shape_signature': array([ 1, -1], dtype=int32), 'dtype': <class 'numpy.int32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}} {'name': 'attention_mask', 'index': 1, 'shape': array([1, 1], dtype=int32), 'shapesignature': array([ 1, -1], dtype=int32), 'dtype': <class 'numpy.bool'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}} {'name': 'speaker_ids', 'index': 2, 'shape': array([1], dtype=int32), 'shape_signature': array([1], dtype=int32), 'dtype': <class 'numpy.int32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}} {'name': 'speed_ratios', 'index': 3, 'shape': array([1], dtype=int32), 'shape_signature': array([1], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}} {'name': 'f0_ratios', 'index': 4, 'shape': array([1], dtype=int32), 'shape_signature': array([1], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}} {'name': 'energy_ratios', 'index': 5, 'shape': array([1], dtype=int32), 'shape_signature': array([1], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}} {'name': 'Identity', 'index': 1149, 'shape': array([ 1, 1, 80], dtype=int32), 'shape_signature': array([-1, -1, 80], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}} {'name': 'Identity_1', 'index': 1185, 'shape': array([ 1, 1, 80], dtype=int32), 'shape_signature': array([-1, -1, 80], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}} {'name': 'Identity_2', 'index': 745, 'shape': array([1, 1], dtype=int32), 'shape_signature': array([-1, -1], dtype=int32), 'dtype': <class 'numpy.int32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}} {'name': 'Identity_3', 'index': 722, 'shape': array([1, 1], dtype=int32), 'shape_signature': array([-1, -1], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}} {'name': 'Identity_4', 'index': 670, 'shape': array([1, 1], dtype=int32), 'shape_signature': array([-1, -1], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}} Traceback (most recent call last): File "inference.py", line 74, in synthesis(text, interpreter, input_details, output_details) File "inference.py", line 62, in synthesis decoder_output_tflite, mel_output_tflite = inference(input_text, interpreter, input_details, output_details) File "inference.py", line 48, in inference interpreter.allocate_tensors() File "anaconda3/lib/python3.7/site-packages/tensorflow/lite/python/interpreter.py", line 231, in allocate_tensors return self._interpreter.AllocateTensors() RuntimeError: tensorflow/lite/kernels/expand_dims.cc:42 axis <= input_dims.size was not true.Node number 0 (EXPAND_DIMS) failed to prepare.

[dathudeptrai]

@ZhaoZeqing can you provide colab notebook so i can easily debug :))).

[ZhaoZeqing]

an you provide colab notebook so i can easily debug :)

Sorry, I'm not familiar with notebook. Can it be converted to a .pb model ?

[dathudeptrai]

@ZhaoZeqing what is ur version of TF, you should install tf-nightly

[ZhaoZeqing]

This is my TF and tf-nightly version:

tensorflow-gpu 2.2.0 tf-estimator-nightly 2.4.0.dev2020071901 tf-nightly 2.3.0.dev20200523

[dathudeptrai]

@ZhaoZeqing did u use an old tflite file, in the newest code i remove attention_mask but i see ur tflite have it, let see:

{'name': 'attention_mask', 'index': 1, 'shape': array([1, 1], dtype=int32), 'shape_signature': array([ 1, -1], dtype=int32), 'dtype': <class 'numpy.bool_'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([],

[ZhaoZeqing]

remove

Thanks for your quickly reply, where is the newest code? I used the code here

[dathudeptrai]

@ZhaoZeqing use newest code on github also. You can see on that colab, we don't see attention_mask anymore.

[ZhaoZeqing]

I'll try the newest code. Thanks!