TFLite support for FastSpeech and Tacotron-2

[dathudeptrai]

Hi, good news, now TensorflowTTS support convert all models supported to TFlite (colab). Thank @jaeyoo from TFlite team for his support. The TTS's game has changed now :D. Let discuss here.

[osamasabri]

thank you for this repo i tried using tacotron.tflite on android i got this error E/AndroidRuntime: FATAL EXCEPTION: main Process: com.example.arabic_tts, PID: 27703 java.lang.RuntimeException: Unable to start activity ComponentInfo{com.example.arabic_tts/com.example.arabic_tts.MainActivity}: java.lang.RuntimeException: java.lang.IllegalArgumentException: ByteBuffer is not a valid flatbuffer model

do you know what tf-nightly version i need to use on android

[dathudeptrai]

@osamasabri you need remove the tf-nightly on android in cache dir :)) then re-install it by this code:

implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly'

[khanhlvg]

You'll also need to add select ops to your dependencies. https://www.tensorflow.org/lite/guide/ops_select#android_aar

[mapledxf]

Thx for the work! But the inference speed is really slow. It takes about 20 seconds to inference the demo input on an Android device. Is there any way to speed up the inference?

[dathudeptrai]

@mapledxf sorry because this mis-understanding. The demo convert Model to 8bit so it run very slow on desktop x86. If you want to check the real latency, pls bring it on real mobile devices (not virtual) or don't convert it to 8bit (keep 32bit) by comment out:

converter.optimizations = [tf.lite.Optimize.DEFAULT]

[mapledxf]

@dathudeptrai Thx for the reply. I am actually running the module on a real Android device (Mi Pad 4).

The following is the test result:

Inference time on a desktop: (without converter.optimizations = [tf.lite.Optimize.DEFAULT]) Tacotron2: 5.997615814208984 Fastspeech: 0.932093620300293 Fastspeech2: 0.9338972568511963

Inference time on Android device: (with converter.optimizations = [tf.lite.Optimize.DEFAULT]) Tacotron2: 19.356

Inference time on Android device: (without converter.optimizations = [tf.lite.Optimize.DEFAULT]) Tacotron2: 24.110

Any idea why?

I haven't tried to port Fastspeech module on to Android device. I will let you know the result once I have done that.

[dathudeptrai]

@mapledxf because Tacotron-2 have dynamic input but fixed output :)), if i remember correctly, it's 1000 frame output of Tacotron2 :v. But note that you should run it some times to get the fastest speed, the first time it run very slow cause by @tf.function. In the future, i think Pruning + fast sparse inference can speed up the latency much, but i don't know when TFlite release it. For now i saw xnnpack but it's still experimental (https://github.com/google/XNNPACK). Maybe you can ask TFlite team to know further trick to optimize latency @jaeyoo

[mapledxf]

@dathudeptrai Hi there, I am trying to make Fastspeech2 tflite working on Android device. However I found that Android's tflite nightly version doesn't not support tf.bool . Any workaround for that?

[dathudeptrai]

@mapledxf ignore attention_mask on input, you can replace it by tf.math.not_equal(input_ids, 0) inside the function.

[mapledxf]

I will try that. Thx!

[mapledxf]

I got 2 seconds inference time with Fastspeech2 on Android device.

GREATE JOB HERE!

[dathudeptrai]

@mapledxf :)) so, let give us a star share our work with ur friend :)), and don't forget to Watch our repo, we will support more :))

[mapledxf]

Any plan to convert the MelGan to TFLite? :)

[manmay-nakhashi]

@mapledxf https://github.com/TensorSpeech/TensorflowTTS/blob/bddd8a70309924adebae7a50e9ed794ec00a5410/notebooks/TensorFlowTTS_FastSpeech%2BMBMelgan_TFLite.ipynb

[mapledxf]

@manmay-nakhashi Thx! It works, but the speed is really slow on Android device.

So I am trying to synthesize with Griffin Lim. def griffin_lim_lb( mel_spec, stats_path, dataset_config, n_iter=32, output_dir=None, wav_name="lb" ):

Where can I get the stats_path for a pre-trained module, i.e. Fastspeech2?

[miyoungvkim]

Hi guys, Is there any plan to share example about tflite inference in android? I'm trying to inference with tensorflow2.tflite in android studio, but I can't find out how to do it.

[mapledxf]

@miyoungvkim I have created a PR. So you can check it out or just wait for merged.

[miyoungvkim]

@mapledxf Thank you very much :)

[dathudeptrai]

I think this issue can be closed since all models support tflite and we already have a real android example. :D.