synthesis with CPU is broken

[roedoejet]

synthesis with a cpu seems to not work: everyvoice synthesize text-to-wav -a cpu -t "This is a test" -v ./logs_and_checkpoints/VocoderExperiment/base/checkpoints/last.ckpt ./logs_and_checkpoints/FeaturePredictionExperiment/base/checkpoints/last.ckpt

Looks like we need to adjust the line where we load the vocoder checkpoint to map to CPU if cuda is not available.

(EveryVoice_lt2) [U20-GPSC5]:$ everyvoice synthesize text-to-wav -a cpu   -t "This is a test"  -v ./logs_and_checkpoints/VocoderExperiment/base/checkpoints/last.ckpt ./logs_and_checkpoints/FeaturePredictionExperiment/base/checkpoints/last.ckpt
/home/tes001/u/TxT2SPEECH/miniconda3_u20/envs/EveryVoice_lt2/lib/python3.9/site-packages/torch/cuda/__init__.py:546: UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")
2023-10-30 16:27:19.295 | INFO     | everyvoice.model.feature_prediction.FastSpeech2_lightning.fs2.cli:synthesize:399 - Loading checkpoint from logs_and_checkpoints/FeaturePredictionExperiment/base/checkpoints/last.ckpt
2023-10-30 16:27:21.545 | INFO     | everyvoice.model.feature_prediction.FastSpeech2_lightning.fs2.cli:synthesize:417 - Processing text 'This is a test'
2023-10-30 16:27:21.546 | INFO     | everyvoice.model.feature_prediction.FastSpeech2_lightning.fs2.cli:synthesize:422 - Creating batch
2023-10-30 16:27:21.547 | INFO     | everyvoice.model.feature_prediction.FastSpeech2_lightning.fs2.cli:synthesize:437 - Predicting spectral features
2023-10-30 16:27:21.982 | INFO     | everyvoice.model.feature_prediction.FastSpeech2_lightning.fs2.cli:synthesize:472 - Loading Vocoder from logs_and_checkpoints/VocoderExperiment/base/checkpoints/last.ckpt
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /gpfs/fs5/nrc/nrc-fs1/ict/others/u/tes001/TxT2SPEECH/EveryVoice_lt2/everyvoice/model/feature_pre │
│ diction/FastSpeech2_lightning/fs2/cli.py:475 in synthesize                                       │
│                                                                                                  │
│   472 │   │   │   │   logger.info(                                                               │
│   473 │   │   │   │   │   f"Loading Vocoder from {model.config.training.vocoder_path}"           │
│   474 │   │   │   │   )                                                                          │
│ ❱ 475 │   │   │   │   ckpt = torch.load(model.config.training.vocoder_path)  # type: ignore      │
│   476 │   │   │   │   logger.info("Generating waveform...")                                      │
│   477 │   │   │   │   wav, sr = synthesize_data(spec, ckpt)                                      │
│   478 │   │   │   │   logger.info(f"Writing file {data_path}")                                   │
│                                                                                                  │
│ /home/tes001/u/TxT2SPEECH/miniconda3_u20/envs/EveryVoice_lt2/lib/python3.9/site-packages/torch/s │
│ erialization.py:809 in load                                                                      │
│                                                                                                  │
│    806 │   │   │   │   │   │   return _load(opened_zipfile, map_location, _weights_only_unpickl  │
│    807 │   │   │   │   │   except RuntimeError as e:                                             │
│    808 │   │   │   │   │   │   raise pickle.UnpicklingError(UNSAFE_MESSAGE + str(e)) from None   │
│ ❱  809 │   │   │   │   return _load(opened_zipfile, map_location, pickle_module, **pickle_load_  │
│    810 │   │   if weights_only:                                                                  │
│    811 │   │   │   try:                                                                          │
│    812 │   │   │   │   return _legacy_load(opened_file, map_location, _weights_only_unpickler,   │
│                                                                                                  │
│ /home/tes001/u/TxT2SPEECH/miniconda3_u20/envs/EveryVoice_lt2/lib/python3.9/site-packages/torch/s │
│ erialization.py:1172 in _load                                                                    │
│                                                                                                  │
│   1169 │                                                                                         │
│   1170 │   unpickler = UnpicklerWrapper(data_file, **pickle_load_args)                           │
│   1171 │   unpickler.persistent_load = persistent_load                                           │
│ ❱ 1172 │   result = unpickler.load()                                                             │
│   1173 │                                                                                         │
│   1174 │   torch._utils._validate_loaded_sparse_tensors()                                        │
│   1175                                                                                           │
│                                                                                                  │
│ /home/tes001/u/TxT2SPEECH/miniconda3_u20/envs/EveryVoice_lt2/lib/python3.9/site-packages/torch/s │
│ erialization.py:1142 in persistent_load                                                          │
│                                                                                                  │
│   1139 │   │   │   typed_storage = loaded_storages[key]                                          │
│   1140 │   │   else:                                                                             │
│   1141 │   │   │   nbytes = numel * torch._utils._element_size(dtype)                            │
│ ❱ 1142 │   │   │   typed_storage = load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location  │
│   1143 │   │                                                                                     │
│   1144 │   │   return typed_storage                                                              │
│   1145                                                                                           │
│                                                                                                  │
│ /home/tes001/u/TxT2SPEECH/miniconda3_u20/envs/EveryVoice_lt2/lib/python3.9/site-packages/torch/s │
│ erialization.py:1116 in load_tensor                                                              │
│                                                                                                  │
│   1113 │   │   # TODO: Once we decide to break serialization FC, we can                          │
│   1114 │   │   # stop wrapping with TypedStorage                                                 │
│   1115 │   │   typed_storage = torch.storage.TypedStorage(                                       │
│ ❱ 1116 │   │   │   wrap_storage=restore_location(storage, location),                             │
│   1117 │   │   │   dtype=dtype,                                                                  │
│   1118 │   │   │   _internal=True)                                                               │
│   1119                                                                                           │
│                                                                                                  │
│ /home/tes001/u/TxT2SPEECH/miniconda3_u20/envs/EveryVoice_lt2/lib/python3.9/site-packages/torch/s │
│ erialization.py:217 in default_restore_location                                                  │
│                                                                                                  │
│    214                                                                                           │
│    215 def default_restore_location(storage, location):                                          │
│    216 │   for _, _, fn in _package_registry:                                                    │
│ ❱  217 │   │   result = fn(storage, location)                                                    │
│    218 │   │   if result is not None:                                                            │
│    219 │   │   │   return result                                                                 │
│    220 │   raise RuntimeError("don't know how to restore data location of "                      │
│                                                                                                  │
│ /home/tes001/u/TxT2SPEECH/miniconda3_u20/envs/EveryVoice_lt2/lib/python3.9/site-packages/torch/s │
│ erialization.py:182 in _cuda_deserialize                                                         │
│                                                                                                  │
│    179                                                                                           │
│    180 def _cuda_deserialize(obj, location):                                                     │
│    181 │   if location.startswith('cuda'):                                                       │
│ ❱  182 │   │   device = validate_cuda_device(location)                                           │
│    183 │   │   if getattr(obj, "_torch_load_uninitialized", False):                              │
│    184 │   │   │   with torch.cuda.device(device):                                               │
│    185 │   │   │   │   return torch.UntypedStorage(obj.nbytes(), device=torch.device(location))  │
│                                                                                                  │
│ /home/tes001/u/TxT2SPEECH/miniconda3_u20/envs/EveryVoice_lt2/lib/python3.9/site-packages/torch/s │
│ erialization.py:166 in validate_cuda_device                                                      │
│                                                                                                  │
│    163 │   device = torch.cuda._utils._get_device_index(location, True)                          │
│    164 │                                                                                         │
│    165 │   if not torch.cuda.is_available():                                                     │
│ ❱  166 │   │   raise RuntimeError('Attempting to deserialize object on a CUDA '                  │
│    167 │   │   │   │   │   │      'device but torch.cuda.is_available() is False. '              │
│    168 │   │   │   │   │   │      'If you are running on a CPU-only machine, '                   │
│    169 │   │   │   │   │   │      'please use torch.load with map_location=torch.device(\'cpu\'  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with
map_location=torch.device('cpu') to map your storages to the CPU.

[marctessier]

Using the latest from main branch.

Yay, this is resolved now. For example I can run this on the "head node" when lazy on small things or send it on a job using a GPU also without needing to select --accelerator ( auto )

Closing this issue