multi-gpu support

[rbychn]

Running the streamlit UI and the program is only using 1 GPU, any way to fix this?

[rbychn]

Text too long error? Is there a way to fix it

[152334H]

Running the streamlit UI and the program is only using 1 GPU, any way to fix this?

I do not currently have access to a multi-gpu setup; it will be difficult for me to personally test any multi-gpu code I could attempt to add.

Text too long error? Is there a way to fix it

Are you using do_tts.py, read.py, or app.py?

[rbychn]

Are you using do_tts.py, read.py, or app.py?

I'm trying it in the textbox provided by the streamlit UI

Traceback (most recent call last): File "C:\ProgramData\Anaconda3\envs\tts-fast\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 565, in _run_script exec(code, module.__dict__) File "C:\Users\Administrator\Desktop\tortoise-tts-fast\app.py", line 151, in <module> gen, dbg_state = tts.tts_with_preset( File "C:\Users\Administrator\Desktop\tortoise-tts-fast\tortoise\api.py", line 376, in tts_with_preset return self.tts(text, **settings) File "C:\Users\Administrator\Desktop\tortoise-tts-fast\tortoise\api.py", line 441, in tts assert text_tokens.shape[-1] < 400, 'Too much text provided. Break the text up into separate segments and re-try inference.' AssertionError: Too much text provided. Break the text up into separate segments and re-try inference. mode 0

[152334H]

@rbychn I just updated the UI with better text splitting capabilities; please pull the repo again and retry

[rbychn]

Long text error is fixed, thanks!

I do not currently have access to a multi-gpu setup; it will be difficult for me to personally test any multi-gpu code I could attempt to add.

If it's not complex, could you please create another branch maybe? I can help with the test.

[152334H]

Honestly I have no idea how to implement multigpu inference, I've never done it before

Unless someone else knows how to do it, I will probably keep this issue open for a while

[rbychn]

OutOfMemoryError: CUDA out of memory. Tried to allocate....If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Is there a way I can edit some files such that I can add the following command?

set CUDA_VISIBLE_DEVICES=0,1 [1]

[rbychn]

Found https://github.com/neonbjb/tortoise-tts/discussions/75 which might be helpful, how would this work when running the streamlit UI command?

[152334H]

Found neonbjb#75 which might be helpful, how would this work when running the streamlit UI command?

the linked discussion describes how to run multiple sessions of tortoise in parallel, i.e. how to run tortoise script A and tortoise script B as separate processes on separate GPUs. It does not describe how to run a single tortoise model on multiple GPUs.

[rbychn]

Sorry, I misunderstood, however, this seems similar?

I'll try doing this and run the UI 2 times, pointing them to different CUDA devices.

[eloop001]

Sorry, I misunderstood, however, this seems similar?

I'll try doing this and run the UI 2 times, pointing them to different CUDA devices.

I have had great luck doing just that. I simply split my input text into two files. Just copy the voice you want to use into two directories nameofvoice1, nameofvoice2 and choose the individual voices for the calls, with the --device argument being 0 for the first and 1 for the second. Call them in separate SSH sessions. It will put the output in separate folders, so one is not overwriting the other.

[Afiyetolsun]

Check my fork