TTS0007: Finding solutions to making TTS faster.

[gangagyatso4364]

Description

We are currently facing issue with speed of inference of tts model. we have two tts models trained mms tts and speech t5. Possible Solutions we should look into.

1. Quantization of speech t5 model.
2. A faster api with model inference from monlam ai web page(could be higher GPU resources).
3. Streaming of TTS inference segment wise in monlam ai web page to make inference appear faster.

Completion Criteria

A faster and precise TTS model for our TTS task in web interface.

[10zinten]

1. Quantization of speech t5 model.

• Downsizing from 32bit to 16bit didn't make any difference in speed and accuracy
• Quantizating with out of the box solution like https://github.com/huggingface/optimum-quanto
  • accuracy: Low
  • solutions
    • Calibration
    • Quantization-Aware-Training
  • speed: Low
  • causes
    • due to overhead of converting int8 weights back to fp36 for inference computation
    • matrix multiplication needs high precision like fp36
  • Conclusion: There is no out of the box solution for quantizing speecht5 model.
• Converted to ONNX format, but no speed gain in CPU and no deployment support for ONNX runtime in huggingface inference API