search

erew123 / alltalk_tts

AllTalk is based on the Coqui TTS engine, similar to the Coqui_tts extension for Text generation webUI, however supports a variety of advanced features, such as a settings page, low VRAM support, DeepSpeed, narrator, model finetuning, custom models, wav file maintenance. It can also be used with 3rd Party software via JSON calls.
GNU Affero General Public License v3.0
1.05k stars 113 forks source link

CUDA not detected/python errors when running finetune/diagnostics also throwing error #323

Closed yaboijbigs closed 2 months ago

yaboijbigs commented 2 months ago

🔴 If you have installed AllTalk in a custom Python environment, I will only be able to provide limited assistance/support. AllTalk draws on a variety of scripts and libraries that are not written or managed by myself, and they may fail, error or give strange results in custom built python environments.

🔴 Please generate a diagnostics report and upload the "diagnostics.log" as this helps me understand your configuration.

https://github.com/erew123/alltalk_tts/tree/main?#-how-to-make-a-diagnostics-report-file

Describe the bug

When I try to run a diagnostics, I get this error: 


  File "C:\Users\bigbe\OneDrive\Desktop\JBigs\clockvideo\alltalk_tts\diagnostics.py", line 388, in <module>
    log_system_info()
  File "C:\Users\bigbe\OneDrive\Desktop\JBigs\clockvideo\alltalk_tts\diagnostics.py", line 188, in log_system_info
    disk_info = get_disk_info()
                ^^^^^^^^^^^^^^^
  File "C:\Users\bigbe\OneDrive\Desktop\JBigs\clockvideo\alltalk_tts\diagnostics.py", line 114, in get_disk_info
    usage = psutil.disk_usage(p.mountpoint)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bigbe\OneDrive\Desktop\JBigs\clockvideo\alltalk_tts\alltalk_environment\env\Lib\site-packages\psutil\__init__.py", line 2040, in disk_usage
    return _psplatform.disk_usage(path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bigbe\OneDrive\Desktop\JBigs\clockvideo\alltalk_tts\alltalk_environment\env\Lib\site-packages\psutil\_pswindows.py", line 290, in disk_usage
    total, free = cext.disk_usage(path)
                  ^^^^^^^^^^^^^^^^^^^^^
OSError: [WinError 1117] The request could not be performed because of an I/O device error: 'D:\\'

When I try to run finetune, I get this error: 

File "C:\Users\bigbe\OneDrive\Desktop\JBigs\clockvideo\alltalk_tts\finetune.py", line 293, in <module>
    system_ram_results = get_system_ram_markdown()
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bigbe\OneDrive\Desktop\JBigs\clockvideo\alltalk_tts\finetune.py", line 247, in get_system_ram_markdown
    system_ram_markdown = f"""
                          ^^^^
ValueError: Unknown format code 'f' for object of type 'str'

When I run start, it says CUDA isn't detected, despite nvcc --version giving me a valid 11.8. :

C:\Users\bigbe\OneDrive\Desktop\JBigs\clockvideo\alltalk_tts\alltalk_environment\env\Lib\site-packages\local_attention\rotary.py:33: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  @autocast(enabled = False)
C:\Users\bigbe\OneDrive\Desktop\JBigs\clockvideo\alltalk_tts\alltalk_environment\env\Lib\site-packages\local_attention\rotary.py:55: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  @autocast(enabled = False)
C:\Users\bigbe\OneDrive\Desktop\JBigs\clockvideo\alltalk_tts\alltalk_environment\env\Lib\site-packages\deepspeed\runtime\zero\linear.py:47: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @autocast_custom_fwd
C:\Users\bigbe\OneDrive\Desktop\JBigs\clockvideo\alltalk_tts\alltalk_environment\env\Lib\site-packages\deepspeed\runtime\zero\linear.py:66: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
  @autocast_custom_bwd
No supported Nvidia GPU found
Using FP32 config instead of FP16 due to GPU compatibility (None)
[AllTalk ENG] Transcoding       : ffmpeg found
[AllTalk ENG] DeepSpeed version : Not available
[AllTalk ENG] Python Version    : 3.11.0
[AllTalk ENG] PyTorch Version   : 2.4.0+cpu
[AllTalk ENG] CUDA Version      : Not available
[AllTalk ENG]
[AllTalk ENG] Model/Engine : xttsv2_2.0.3 loading into cpu

To Reproduce I installed the standalone version and followed all the instructions to a tee.

Screenshots Code attached above

Text/logs If applicable, copy/paste in your logs here from the console.

See above

Desktop (please complete the following information): AllTalk was updated: [approx. date] Custom Python environment: [yes/no give details if yes] Text-generation-webUI was updated: [approx. date]

Additional context Add any other context about the problem here.

Side note: sorry if this is the incorrect way to file an issue -- this is my first time doing it. Happy to help and do anything needed to resolve :)

erew123 commented 2 months ago

Hi @yaboijbigs

Its fair to say that your Python Environment isnt correct, specifically Pytorch. This is shown here:

image

nvcc --version is showing the CUDA Development Toolkit version, not the Pytorch CUDA version. These are 2x separate things and therefore not directly related to one another.

OSError: [WinError 1117] The request could not be performed because of an I/O device error: 'D:\

I have no idea why your D drive is inaccessible/has an I/O error, but this is an issue reported by Windows. Ive updated the diagnostics to work around that.

ValueError: Unknown format code 'f' for object of type 'str'

I would assume this is back to what I mentioned earlier about the Python environment not being correct.

C:\Users\bigbe\OneDrive\Desktop\JBigs\clockvideo\alltalk_tts\

I see you are using Onedrive, this could be causing you issues storing such large amounts of information this way. Onedrive can remove files from your computer, to only be stored in the cloud unless called back to disk. I cannot say IF such a system of pulling files back from the cloud would or wouldnt work in a timely way. AllTalk runs on lots of code that is not written by myself and I am unable to validate all their code. Please see here for a reference about Onedrive behaviour https://support.microsoft.com/en-gb/office/use-onedrive-and-storage-sense-in-windows-10-to-manage-disk-space-de5faa9a-6108-4be1-87a6-d90688d08a48

General approach to resolve

Because of the various things above, I can only give you general guidance. Suggestions are:

1) Open a command prompt and run the atsetup.bat 2) Git Pull the updates, Delete the AllTalk environment, Purge the Pip cache and re-setup the Python Environment.

image

3) Watch out for errors during installation. Once complete, you can try again and you would at least be able to run the diagnostics if there is an issue.

As mentioned, there may be issues storing things like this withing a Onedrive folder location, which you may need to investigate separately, along with figuring out why your D drive is inaccessible (though D drive being inaccessible is not causing AllTalk any issues and the diagnostics should now run irrespective).

Thanks