Error training variance model

openvpi / DiffSinger

An advanced singing voice synthesis system with high fidelity, expressiveness, controllability and flexibility based on DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism

Apache License 2.0

2.73k stars 288 forks source link

Error training variance model #199

Closed Surya-29 closed 4 months ago

Surya-29 commented 5 months ago

I'm getting this error during the start of variance model training. I also tried changing the configuration parameters (all pl configs and also tried setting nccl_p2p : false ) but yet the error keeps popping up.

Epoch 0: 0%| | 0/30 [00:00<?, ?it/s]terminate called after throwing an instance of 'std::runtime_error' what(): torch_xla/csrc/runtime/runtime.cc:23 : $PJRT_DEVICE is not set.

KakaruHayate commented 4 months ago

torch_xla... You are running on TPU?

Surya-29 commented 4 months ago

Nope, I'm running it on a GCP VM and the compute engine is T4 GPU

ghost commented 4 months ago

Are you running GCP VM with Google Colab Notebook? I have seen Colab preinstalled (and mismatch) dependency. Maybe you have to try on a "fresh" system, with the maintainer provided requirements installed.