Turning `model.learn_alignment=false` breaks `everyvoice preprocess`

[SamuelLarkin]

How to reproduce

• run everyvoice wizard
• run preprocessing disabling learn_alignment everyvoice preprocess config/everyvoice-text-to-spec.yaml -c model.learn_alignment=false

Log

2024-06-05 09:52:40.124 | INFO     | everyvoice.config.validation_helpers:directory_path_must_exist:89 - Directory at /gpfs/fs3c/nrc/dt/sam037/exp/EveryVoice/tiny.lj/resume/preprocessed does not exist. Creating...
2024-06-05 09:52:40.126 | INFO     | everyvoice.utils:update_config_from_cli_args:170 - Updating config 'model.learn_alignment' to value 'false'
2024-06-05 09:52:40.127 | INFO     | everyvoice.config.utils:load_partials:47 - You have both the key preprocessing and path_to_preprocessing_config_file defined in your configuration. We will override values from path_to_preprocessing_config_file with values from preprocessing
2024-06-05 09:52:40.130 | INFO     | everyvoice.config.utils:load_partials:47 - You have both the key text and path_to_text_config_file defined in your configuration. We will override values from path_to_text_config_file with values from text
^MDataset:   0%|          | 0/1 [00:00<?, ?it/s]2024-06-05 09:52:40.449 | INFO     | everyvoice.preprocessor.preprocessor:process_all_audio:515 - Collecting files for dataset_0 and processing audio
2024-06-05 09:52:40.449 | INFO     | everyvoice.preprocessor.preprocessor:process_all_audio:519 - Launching parallel processes may take a moment...
type                        quantity
--------------------------  ----------
processed files             500
previously processed files  0
missing files               0
missing symbols             0
skipped processes           0
nans                        0
audio_empty                 0
audio_too_short             0
audio_too_long              0
duration                    0h 54m 31s
2024-06-05 09:53:38.186 | INFO     | everyvoice.preprocessor.preprocessor:preprocess:981 - Processing spec on 4 CPUs...
2024-06-05 09:54:08.783 | INFO     | everyvoice.preprocessor.preprocessor:preprocess:981 - Processing attn on 4 CPUs...
2024-06-05 09:54:30.245 | INFO     | everyvoice.preprocessor.preprocessor:preprocess:981 - Processing energy on 4 CPUs...
_RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/sam037/.conda/envs/EveryVoice.sl/lib/python3.11/site-packages/joblib/externals/loky/process_executor.py", line 463, in _process_worker
    r = call_item()
        ^^^^^^^^^^^
  File "/home/sam037/.conda/envs/EveryVoice.sl/lib/python3.11/site-packages/joblib/externals/loky/process_executor.py", line 291, in __call__
    return self.fn(*self.args, **self.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sam037/.conda/envs/EveryVoice.sl/lib/python3.11/site-packages/joblib/parallel.py", line 589, in __call__
    return [func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sam037/.conda/envs/EveryVoice.sl/lib/python3.11/site-packages/joblib/parallel.py", line 589, in <listcomp>
    return [func(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/fs/hestia_Hnrc/ict/sam037/git/EveryVoice/everyvoice/preprocessor/preprocessor.py", line 563, in process_energy
    durs = torch.load(dur_path)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/sam037/.conda/envs/EveryVoice.sl/lib/python3.11/site-packages/torch/serialization.py", line 986, in load
    with _open_file_like(f, 'rb') as opened_file:
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sam037/.conda/envs/EveryVoice.sl/lib/python3.11/site-packages/torch/serialization.py", line 435, in _open_file_like
    return _open_file(name_or_buffer, mode)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sam037/.conda/envs/EveryVoice.sl/lib/python3.11/site-packages/torch/serialization.py", line 416, in __init__
    super().__init__(open(name, mode))
                     ^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/gpfs/fs3c/nrc/dt/sam037/exp/EveryVoice/tiny.lj/resume/preprocessed/duration/LJ001-0001--default--eng--duration.pt'
"""

The above exception was the direct cause of the following exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /fs/hestia_Hnrc/ict/sam037/git/EveryVoice/everyvoice/model/feature_prediction/FastSpeech2_lightn │
│ ing/fs2/cli/preprocess.py:35 in preprocess                                                       │
│                                                                                                  │
│   32 │                                                                                           │
│   33 │   from ..config import FastSpeech2Config                                                  │
│   34 │                                                                                           │
│ ❱ 35 │   preprocessor, config, processed = preprocess_base_command(                              │
│   36 │   │   model_config=FastSpeech2Config,                                                     │
│   37 │   │   steps=[step.name for step in steps],                                                │
│   38 │   │   **kwargs,                                                                           │
│                                                                                                  │
│ /fs/hestia_Hnrc/ict/sam037/git/EveryVoice/everyvoice/base_cli/helpers.py:117 in                  │
│ preprocess_base_command                                                                          │
│                                                                                                  │
│   114 │   │   == TargetTrainingTextRepresentationLevel.phonological_features                     │
│   115 │   ):                                                                                     │
│   116 │   │   steps.append("pfs")                                                                │
│ ❱ 117 │   preprocessor.preprocess(                                                               │
│   118 │   │   cpus=cpus,                                                                         │
│   119 │   │   overwrite=overwrite,                                                               │
│   120 │   │   to_process=steps,                                                                  │
│                                                                                                  │
│ /fs/hestia_Hnrc/ict/sam037/git/EveryVoice/everyvoice/preprocessor/preprocessor.py:991 in         │
│ preprocess                                                                                       │
│                                                                                                  │
│    988 │   │   │   │   │   │   │   total=len(filelist),                                          │
│    989 │   │   │   │   │   │   )                                                                 │
│    990 │   │   │   │   │   ):                                                                    │
│ ❱  991 │   │   │   │   │   │   Parallel(                                                         │
│    992 │   │   │   │   │   │   │   n_jobs=self.cpus,                                             │
│    993 │   │   │   │   │   │   │   backend="loky",                                               │
│    994 │   │   │   │   │   │   │   batch_size=batch_size,                                        │
│                                                                                                  │
│ /home/sam037/.conda/envs/EveryVoice.sl/lib/python3.11/site-packages/joblib/parallel.py:1952 in   │
│ __call__                                                                                         │
│                                                                                                  │
│   1949 │   │   # dispatch of the tasks to the workers.                                           │
│   1950 │   │   next(output)                                                                      │
│   1951 │   │                                                                                     │
│ ❱ 1952 │   │   return output if self.return_generator else list(output)                          │
│   1953 │                                                                                         │
│   1954 │   def __repr__(self):                                                                   │
│   1955 │   │   return '%s(n_jobs=%s)' % (self.__class__.__name__, self.n_jobs)                   │
│                                                                                                  │
│ /home/sam037/.conda/envs/EveryVoice.sl/lib/python3.11/site-packages/joblib/parallel.py:1595 in   │
│ _get_outputs                                                                                     │
│                                                                                                  │
│   1592 │   │   │   yield                                                                         │
│   1593 │   │   │                                                                                 │
│   1594 │   │   │   with self._backend.retrieval_context():                                       │
│ ❱ 1595 │   │   │   │   yield from self._retrieve()                                               │
│   1596 │   │                                                                                     │
│   1597 │   │   except GeneratorExit:                                                             │
│   1598 │   │   │   # The generator has been garbage collected before being fully                 │
│                                                                                                  │
│ /home/sam037/.conda/envs/EveryVoice.sl/lib/python3.11/site-packages/joblib/parallel.py:1699 in   │
│ _retrieve                                                                                        │
│                                                                                                  │
│   1696 │   │   │   # exception (e.g. `GeneratorExit`), exit the loop and surface the             │
│   1697 │   │   │   # worker traceback.                                                           │
│   1698 │   │   │   if self._aborting:                                                            │
│ ❱ 1699 │   │   │   │   self._raise_error_fast()                                                  │
│   1700 │   │   │   │   break                                                                     │
│   1701 │   │   │                                                                                 │
│   1702 │   │   │   # If the next job is not ready for retrieval yet, we just wait for            │
│                                                                                                  │
│ /home/sam037/.conda/envs/EveryVoice.sl/lib/python3.11/site-packages/joblib/parallel.py:1734 in   │
│ _raise_error_fast                                                                                │
│                                                                                                  │
│   1731 │   │   # calling get_result. This job might not exists if abort has been                 │
│   1732 │   │   # called directly or if the generator is gc'ed.                                   │
│   1733 │   │   if error_job is not None:                                                         │
│ ❱ 1734 │   │   │   error_job.get_result(self.timeout)                                            │
│   1735 │                                                                                         │
│   1736 │   def _warn_exit_early(self):                                                           │
│   1737 │   │   """Warn the user if the generator is gc'ed before being consumned."""             │
│                                                                                                  │
│ /home/sam037/.conda/envs/EveryVoice.sl/lib/python3.11/site-packages/joblib/parallel.py:736 in    │
│ get_result                                                                                       │
│                                                                                                  │
│    733 │   │   │   # We assume that the result has already been retrieved by the                 │
│    734 │   │   │   # callback thread, and is stored internally. It's just waiting to             │
│    735 │   │   │   # be returned.                                                                │
│ ❱  736 │   │   │   return self._return_or_raise()                                                │
│    737 │   │                                                                                     │
│    738 │   │   # For other backends, the main thread needs to run the retrieval step.            │
│    739 │   │   try:                                                                              │
│                                                                                                  │
│ /home/sam037/.conda/envs/EveryVoice.sl/lib/python3.11/site-packages/joblib/parallel.py:754 in    │
│ _return_or_raise                                                                                 │
│                                                                                                  │
│    751 │   def _return_or_raise(self):                                                           │
│    752 │   │   try:                                                                              │
│    753 │   │   │   if self.status == TASK_ERROR:                                                 │
│ ❱  754 │   │   │   │   raise self._result                                                        │
│    755 │   │   │   return self._result                                                           │
│    756 │   │   finally:                                                                          │
│    757 │   │   │   del self._result                                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [Errno 2] No such file or directory: '/gpfs/fs3c/nrc/dt/sam037/exp/EveryVoice/tiny.lj/resume/preprocessed/duration/LJ001-0001--default--eng--duration.pt'

[roedoejet]

This is breaking for a similar reason described in https://github.com/roedoejet/EveryVoice/issues/452#issuecomment-2166684559 except it's probably also the fact that you have the pitch_target and energy_target in the config set to 'phones' instead of 'frames', and if learn_alignment is set to False, it will look for the durations instead of calculating the prior that is used by the jointly-learned alignment. This also appears to be working as-expected, but we could consider adding an error message. Honestly, turning this to False is really only intended to be done by advanced users of EveryVoice though, so I'm hesitant to provide too much direction here.