dosubot[bot]
commented
1 month ago The issue you're experiencing with the text-to-speech (TTS) functionality in your self-hosted Dify version 0.6.16 is likely due to the way the TTS model handles long texts. The XinferenceText2SpeechModel class in the tts.py file has a method _tts_invoke_streaming that processes the text for TTS. This method includes a word limit and splits the text into sentences if it exceeds this limit.
To resolve this issue, you can adjust the word limit or modify the way the text is split and processed. Here is the relevant part of the code:
class XinferenceText2SpeechModel(TTSModel):
def _tts_invoke_streaming(self, model: str, credentials: dict, content_text: str, voice: str) -> any:
if credentials['server_url'].endswith('/'):
credentials['server_url'] = credentials['server_url'][:-1]
try:
handle = RESTfulAudioModelHandle(credentials['model_uid'], credentials['server_url'], auth_headers={})
model_support_voice = [x.get("value") for x in self.get_tts_model_voices(model=model, credentials=credentials)]
if not voice or voice not in model_support_voice:
voice = self._get_model_default_voice(model, credentials)
word_limit = self._get_model_word_limit(model, credentials)
if len(content_text) > word_limit:
sentences = self._split_text_into_sentences(content_text, max_length=word_limit)
executor = concurrent.futures.ThreadPoolExecutor(max_workers=min(3, len(sentences)))
futures = [executor.submit(
handle.speech,
input=sentences[i],
voice=voice,
response_format="mp3",
speed=1.0,
stream=False
) for i in range(len(sentences))]
for index, future in enumerate(futures):
response = future.result()
for i in range(0, len(response), 1024):
yield response[i:i + 1024]
else:
response = handle.speech(
input=content_text.strip(),
voice=voice,
response_format="mp3",
speed=1.0,
stream=False
)
for i in range(0, len(response), 1024):
yield response[i:i + 1024]
except Exception as ex:
raise InvokeBadRequestError(str(ex))To accommodate longer texts, you can increase the word limit or improve the text splitting logic. For example, you can set a higher word limit:
def _get_model_word_limit(self, model: str, credentials: dict) -> int:
return 5000 # Increased word limitAdditionally, ensure that the text splitting logic in _split_text_into_sentences is efficient and handles longer texts appropriately. This change should allow the TTS functionality to process and return more complete audio messages for longer texts [1][2].
To continue talking to Dosu, mention @dosu.
401557122
Self Checks
Dify version
0.6.16
Cloud or Self Hosted
Self Hosted (Docker)
Steps to reproduce
I deployed chatTTS using xinference and set it to autoplay on dify. I found that on the agent's debugging page, short sentences can be autoplayed. However, when the generated text is long or occasionally appears, the returned tts message may not be the audio of all the text content, but may only be the first two sentences
✔️ Expected Behavior
No response
❌ Actual Behavior
No response