TTS is stuck with PyTorch 1.12

TareHimself commented 2 years ago

Hi, after i send the tts model some text to convert it gets stuck for a while then prints this

C:\**\PycharmProjects\voice\venv\lib\site-packages\torch\nn\modules\module.py:1130: UserWarning: operator () profile_node %433 : int[] = prim::profile_ivalue(%431) does not have profile information (Triggered internally at ..\torch\csrc\jit\codegen\cuda\graph_fuser.cpp:108.) return forward_call(*input, **kwargs) C:\**\PycharmProjects\voice\venv\lib\site-packages\torch\nn\modules\module.py:1130: UserWarning: concrete shape for linear input & weight are required to decompose into matmul + bias (Triggered internally at ..\torch\csrc\jit\codegen\cuda\graph_fuser.cpp:2077.) return forward_call(*input, **kwargs)

The model is https://models.silero.ai/models/tts/en/v3_en.pt at 48000 using en_10

snakers4 commented 2 years ago

Hi,

Does this happen with PyTorch 1.12?

TareHimself commented 2 years ago

torch 1.12.1+cu116

snakers4 commented 2 years ago

A quick fix is to use PyTorch 1.11. This happens only with PyTorch 1.12.

We checked extensively why this happens, this boils down to one particular (mostly stadard!) layer in our model. Surprisingly, this was not the case in 1.9, 1.10 and 1.11. This also does not happen with other models with the same layer. Internally, we decided to wait until 1.13, since we could not find an easy fix and we faced other compiler issues with 1.12.

Also doing this may help with 1.11:

torch._C._jit_set_profiling_mode(False)

TareHimself commented 2 years ago

Thanks, that worked

debagos commented 2 years ago

Had the same problem, adding

torch._C._jit_set_profiling_mode(False)

to my code fixed the issue for me as well.

snakers4 commented 2 years ago

to my code fixed the issue for me as well.

Then it's official, JIT compiler is 100% to blame here.

snakers4 / silero-models

TTS is stuck with PyTorch 1.12 #183