search

R3gm / SoniTranslate

Synchronized Translation for Videos. Video dubbing
Apache License 2.0
334 stars 58 forks source link

Speech too fast and out of sync #19

Closed agenceblink closed 1 month ago

agenceblink commented 4 months ago

Hello and congratulations for this work! I just tested a lot of other projects and yours is clearly the most efficient :)

Unfortunately and I do not understand why: the speech of the translation has a speed like x10 and we do not understand anything of course ^^ and then 15 seconds later, the speech speed returns to a correct speed. Here is a short extract to illustrate the problem: FinalTest

Could you give me a clue so that I can find a solution?

(for the same project, I've done a transcription + translate with wisper without any problems, but I'm missing the voice translation and video sync).

Many thanks in advance for your reply!

R3gm commented 4 months ago

Hello,

The TTS generates segments individually and then combines them into a single audio file. Sometimes, the duration of these segments exceeds the expected timeframe, causing overlap. To compensate, the program accelerates some segments based on their timestamps. Whisper may not always detect precise start and end timestamps, leading to accelerated segments when it shouldn't.

To address this issue, you can adjust the maximum speech acceleration for all segments in:

Advanced Settings

Max Audio Acceleration

The default value is 1.9. You can try other values like 1.5 or 1.0 to disable the acceleration. However, be cautious as this may cause some segments of speech to overlap.

Slyfoxleo commented 4 months ago

Thanks for the great work! Is it possible to slow down the video itself so that the audio segment comes in at normal speed? Thank you!

R3gm commented 4 months ago

Thanks for the great work! Is it possible to slow down the video itself so that the audio segment comes in at normal speed? Thank you!

At the moment, that feature isn't available, but I'll keep it in mind for future updates. In the meantime, you can slow down the entire video before processing it to achieve the desired effect.

agenceblink commented 4 months ago

Your answer is as precise as your work! Thanks man :) Question in passing: are you thinking of porting your script to Pinokio?

R3gm commented 4 months ago

I haven't heard of it before, but I'll definitely look into it, thanks!