search

KoljaB / LocalAIVoiceChat

Local AI talk with a custom voice based on Zephyr 7B model. Uses RealtimeSTT with faster_whisper for transcription and RealtimeTTS with Coqui XTTS for synthesis.
Other
479 stars 51 forks source link

Will switching to SeamlessM4Tv2 be better #4

Open jkfnc opened 10 months ago

jkfnc commented 10 months ago

SeamlessM4Tv2 Released today seems to have all this and translation with streaming support ? Will it be better than Whisper and Coqui ?

KoljaB commented 10 months ago

Probably yes. Advancements in this area are made crazy fast, i feel stuff like this expires in like 4 weeks.

jkfnc commented 10 months ago

I think it may have to be a flag between whisper and Meta as Seamless M4T V2 is still CC-BY-NC license which is incompatible with your MIT License.

KoljaB commented 10 months ago

Damn. You are right and this also counts for the coqui. Need to revoke MIT 2 here asap.

jayakumark commented 10 months ago

Probably this one https://huggingface.co/spaces/styletts2/styletts2 can replace Coqui and its MIT.

KoljaB commented 10 months ago

Not sure about that. StyleTTS2 is only good in english and can't zero shot voice clone.

jkfnc commented 9 months ago

This one seems to capture Tone and emotion is what they claim https://research.myshell.ai/open-voice/zero-shot-cross-lingual-voice-cloning may work for your TurnVoice project

stevenbaert commented 7 months ago

Fyi, this is what I found as extra information (Mac):

Use Apple's Metal for GPU Acceleration: Apple provides the Metal framework for GPU acceleration on macOS. Some machine learning libraries, like TensorFlow and PyTorch, offer ways to leverage Metal for acceleration through third-party initiatives or experimental support.

For PyTorch, there's an experimental project called PyTorch-Metal that aims to bring Metal GPU acceleration to PyTorch on macOS. For TensorFlow, you might explore Apple's Metal plugin for TensorFlow that enables using Metal for accelerated machine learning operations.

Use PlaidML: PlaidML is an open-source tensor compiler that can enable deep learning on different types of GPUs, including AMD GPUs found in many Macs. It works with Keras as a backend and can be a way to leverage your Mac's GPU for acceleration.

ajeema commented 6 months ago

maybe MeloTTS/OpenVoice would be a good replacement, also distil_whisper

KoljaB commented 5 months ago

maybe MeloTTS/OpenVoice would be a good replacement, also distil_whisper

You can alread use distil whisper models. Update your faster whisper to latest version (pip install -U faster-whisper), then change the model to one of the distil supported ones (distil-large-v2, distil-medium.en, distil-small.en) in this line:

recorder = AudioToTextRecorder(model="tiny.en", language="en", spinner=False)

Melo I found to have rather bad quality (so few emotions) and OpenVoice is a research project which does not get updates. So I won't implement those into RealtimeTTS (takes a lot for a TTS engine to be considered for me to make it realtime).