search

ThioJoe / Auto-Synced-Translated-Dubs

Automatically translates the text of a video based on a subtitle file, and then uses AI voice services to create a new dubbed & translated audio track where the speech is synced using the subtitle's timings.
GNU General Public License v3.0
1.6k stars 158 forks source link

Empty audio clips #43

Open ilMelle opened 1 year ago

ilMelle commented 1 year ago

The biggest issue, Azure creates empty audio clips, but not all of them. I updated the code according to #17, disabled batch_tts_synthesize and set add_line_buffer_milliseconds to 0. On the side, all the clips are downloaded outside the workingFolder and then never deleted.

Here is a screenshot of the terminal:

Schermata 2023-01-12 alle 13 54 27

And the app folder looks like this afterwards

Schermata 2023-01-12 alle 13 55 06
ThioJoe commented 1 year ago

Can you provide your original srt file?

ilMelle commented 1 year ago

Sure, here they are Tacchette_ITA.srt.zip

ilMelle commented 1 year ago

I solved the "files outside the workfolder" issue, by changing line 332 in TTS.py like this filePath = f"workingFolder/{str(key)}.mp3" since I'm workin con MacOS.

I should also add: that the "empty files" issue seems to be hit or miss: your script works perfectly 20% of the time

ThioJoe commented 1 year ago

Ok I updated some lines so that it doesn't hardcode the paths, but rather uses the os.path.join() function, so that should fix at least the one issue you just mentioned. Try using the latest versions of TTS.py, audio_builder.py, and main.py

I'm not sure if this will fix the issue with the blank audio clips, so let me know.

ilMelle commented 1 year ago

The audio clips appear inside "workingFolder", but it still has a high failure rate: I need to run the script a few times before it creates usable MP3 files for ffmpeg to work with. The empty files are always different, sometimes is the majority and some other times are just two or three. Here a few screenshots.

Schermata 2023-01-13 alle 13 09 09 Schermata 2023-01-13 alle 13 13 02 Schermata 2023-01-13 alle 13 13 42
ThioJoe commented 1 year ago

Are you able to use batch synthesis or did you just disable it as something to try?

Also try using debug mode. Download the latest version of the code here: https://github.com/ThioJoe/Auto-Synced-Translated-Dubs/archive/refs/heads/main.zip

Then in config.ini , set debug_mode = True . This should make it so it will produce copies of the sound files for each step in workingFolder instead of deleting them. If you use batch mode it will be easier, because it will just keep the entire zip file downloaded from Azure in-tact. Then you can look in there to see if the files are indeed empty when you downloaded them, or if it was happening when processing them.

If you aren't using batch mode, debug mode should still work, but the files will be named different. So x_p1.mp3 will be the file downloaded, x_t.mp3 is the first downloaded file trimmed, x_p2.mp3 will be the second pass downloaded, and x_p2_t.mp3 will be the second pass trimmed.

This should let you see if they truly are coming from Azure empty, or if something else is going on.

ilMelle commented 1 year ago

I was able to use batch mode, but only upgrading from the free F0 tier to the paid S0 tier of Azure. I'll try debug mode to check anyways, but it would have been nice to use your (amazing) script without a paid subscription.

I also tried Google voice synthesis, but it displays a different error and I couldn't go forward. Thanks for your help, I hope this tool will get more stable in the future.