Error training variance model

openvpi / DiffSinger

An advanced singing voice synthesis system with high fidelity, expressiveness, controllability and flexibility based on DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism

Apache License 2.0

2.62k stars 275 forks source link

Error training variance model #199

Closed Surya-29 closed 2 days ago

Surya-29 commented 5 days ago

I'm getting this error during the start of variance model training. I also tried changing the configuration parameters (all pl configs and also tried setting nccl_p2p : false ) but yet the error keeps popping up.

Epoch 0: 0%| | 0/30 [00:00<?, ?it/s]terminate called after throwing an instance of 'std::runtime_error' what(): torch_xla/csrc/runtime/runtime.cc:23 : $PJRT_DEVICE is not set.

KakaruHayate commented 5 days ago

torch_xla... You are running on TPU?

Surya-29 commented 5 days ago

Nope, I'm running it on a GCP VM and the compute engine is T4 GPU

tideka commented 4 days ago

Are you running GCP VM with Google Colab Notebook? I have seen Colab preinstalled (and mismatch) dependency. Maybe you have to try on a "fresh" system, with the maintainer provided requirements installed.