Error when using generated FastSpeech2 TFLite model with Android example

[OscarVanL]

Hi, this one's probably for @mapledxf and is a duplicate of this issue: https://github.com/TensorSpeech/TensorFlowTTS/issues/274

I've converted my FastSpeech2 model to a TFLite model in the way shown on Colab.

I've taken the output TFLite model, put it in the assets folder on the Android Example, and renamed it to fastspeech2_quant.tflite to replace the existing fastspeech2 tflite. Then when I type in some text into the box and press speak, an error occurs.

Here's the Android app logs:

Android App error logs

``` D/AudioTrack: ClientUid 10397 AudioTrack::start D/TtsStateDispatcher: onTtsReady: I/AssistStructure: Flattened final assist data: 3320 bytes, containing 1 windows, 14 views D/InputWorker: add to queue: testing testing this is a voice test D/InputWorker: processing: testing testing this is a voice test D/TtsStateDispatcher: onTtsStart: D/InputWorker: speak: [testing testing this is a voice test] D/processor: text preprocessed: testing testing this is a voice test D/FastSpeech2: input id length: 36 E/InputWorker: Exception: java.lang.IllegalArgumentException: Cannot convert between a TensorFlowLite tensor with type FLOAT32 and a Java object of type [I (which is compatible with the TensorFlowLite type INT32). at org.tensorflow.lite.Tensor.throwIfTypeIsIncompatible(Tensor.java:427) at org.tensorflow.lite.Tensor.getInputShapeIfDifferent(Tensor.java:287) at org.tensorflow.lite.NativeInterpreterWrapper.run(NativeInterpreterWrapper.java:146) at org.tensorflow.lite.Interpreter.runForMultipleInputsOutputs(Interpreter.java:360) at com.tensorspeech.tensorflowtts.module.FastSpeech2.getMelSpectrogram(FastSpeech2.java:67) at com.tensorspeech.tensorflowtts.tts.InputWorker$InputText.proceed(InputWorker.java:86) at com.tensorspeech.tensorflowtts.tts.InputWorker$InputText.access$700(InputWorker.java:66) at com.tensorspeech.tensorflowtts.tts.InputWorker.lambda$new$0$InputWorker(InputWorker.java:43) at com.tensorspeech.tensorflowtts.tts.-$$Lambda$InputWorker$jYcArgm2l9qkL5ylkeKfv_zVAG8.run(Unknown Source:2) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:462) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:919) ```

Here's some suspicious logs made when doing the conversion, which may be related:

TFLite Conversion Logs

``` 2020-10-30 23:25:24.454753: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._0/attention/self/Shape that is not type float. 2020-10-30 23:25:24.470916: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._0/attention/self/Shape that is not type float. 2020-10-30 23:25:24.488655: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._0/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.509365: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._0/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.533000: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._1/attention/self/Shape that is not type float. 2020-10-30 23:25:24.553011: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._1/attention/self/Shape that is not type float. 2020-10-30 23:25:24.573864: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._1/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.595381: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._1/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.619833: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._2/attention/self/Shape that is not type float. 2020-10-30 23:25:24.638449: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._2/attention/self/Shape that is not type float. 2020-10-30 23:25:24.657566: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._2/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.677951: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._2/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.698034: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._3/attention/self/Shape that is not type float. 2020-10-30 23:25:24.717519: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._3/attention/self/Shape that is not type float. 2020-10-30 23:25:24.738637: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._3/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.758332: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._3/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.778157: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor duration_predictor/dense_3/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.803053: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor duration_predictor/dense_3/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.822459: I tensorflow/lite/tools/optimize/quantize_weights.cc:211] Skipping quantization of tensor duration_predictor/dense_3/Tensordot/MatMul because it has fewer than 1024 elements (256). 2020-10-30 23:25:24.844940: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor energy_predictor/dense_2/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.868953: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor energy_predictor/dense_2/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.888406: I tensorflow/lite/tools/optimize/quantize_weights.cc:211] Skipping quantization of tensor energy_predictor/dense_2/Tensordot/MatMul because it has fewer than 1024 elements (256). 2020-10-30 23:25:24.909410: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor f0_predictor/dense_1/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.936395: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor f0_predictor/dense_1/Tensordot/Shape that is not type float. 2020-10-30 23:25:24.958783: I tensorflow/lite/tools/optimize/quantize_weights.cc:211] Skipping quantization of tensor f0_predictor/dense_1/Tensordot/MatMul because it has fewer than 1024 elements (256). 2020-10-30 23:25:24.990119: I tensorflow/lite/tools/optimize/quantize_weights.cc:211] Skipping quantization of tensor length_regulator/Repeat/boolean_mask/Reshape because it has fewer than 1024 elements (1). 2020-10-30 23:25:25.020595: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._0/attention/self/Shape that is not type float. 2020-10-30 23:25:25.043419: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._0/attention/self/Shape that is not type float. 2020-10-30 23:25:25.065443: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._0/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:25.090323: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._0/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:25.113152: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._1/attention/self/Shape that is not type float. 2020-10-30 23:25:25.134870: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._1/attention/self/Shape that is not type float. 2020-10-30 23:25:25.155454: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._1/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:25.178455: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._1/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:25.210206: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._2/attention/self/Shape that is not type float. 2020-10-30 23:25:25.240226: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._2/attention/self/Shape that is not type float. 2020-10-30 23:25:25.265304: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._2/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:25.289809: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._2/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:25.317303: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._3/attention/self/Shape that is not type float. 2020-10-30 23:25:25.343394: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._3/attention/self/Shape that is not type float. 2020-10-30 23:25:25.376316: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._3/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:25.405087: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._3/attention/output/dense/Tensordot/Shape that is not type float. 2020-10-30 23:25:25.426959: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor mel_before/Tensordot/Shape that is not type float. 2020-10-30 23:25:25.449968: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor mel_before/Tensordot/Shape that is not type float. Writing converted model to: .\model-215000.tflite ```

Any ideas what's going on here? Thanks :)

[mapledxf]

according to this issue , new version model has removed attention mask as an input. so u should also delete attention mask input for android which is coded here https://github.com/TensorSpeech/TensorFlowTTS/blob/e42595abbf21208c81e0fabaa0b1eaeaca2c4053/examples/android/app/src/main/java/com/tensorspeech/tensorflowtts/module/FastSpeech2.java#L68

[OscarVanL]

Thank you! For anyone that comes across this issue, the new line should become:

new Object[]{inputs, new int[]{0}, new float[]{speed}, new float[]{1F}, new float[]{1F}},

Because the inference inputs are: input_ids, speaker_ids, speed_ratio, f0_ratios, energy_ratios.

This solved the problem, thank you.

[dathudeptrai]

Thank you! For anyone that comes across this issue, the new line should become:

new Object[]{inputs, new int[]{0}, new float[]{speed}, new float[]{1F}, new float[]{1F}},

Because the inference inputs are: input_ids, speaker_ids, speed_ratio, f0_ratios, energy_ratios.

This solved the problem, thank you.

can you make a pull request to fix a android example code :)). There are some reasons that prevent me contribute to android demo :)))

[OscarVanL]

There is one problem...

The existing model in the example probably requires the old input shape, but newly trained models will use the new input shape.

If I fix the example, then it will probably break the fastspeech2_quant.tflite model under TensorFlowTTS/examples/android/app/src/main/assets.

So to do this pull request, we also need a newly trained Fastspeech2 model, I think.

[mapledxf]

@OscarVanL yeah, ur right. That's why i didnt make the PR to correct this.