[BUG] IndexError when training new acoustic model

[massimolipari]

Debugging checklist

[X] Have you updated to latest MFA version? [X] Have you tried rerunning the command with the --clean flag?

Describe the issue When training an acoustic model with the command mfa train --clean --no_textgrid_cleanup --single_speaker --no_debug --no_verbose --no_use_mp <dict_path> <output_path>, an IndexError is thrown at the very beginning of the Initializing training for monophone... step.

For Reproducing your issue Please fill out the following:

1. Corpus structure
   • What language is the corpus in? French (Quebec)
   • How many files/speakers? 550 files, 62 speakers
   • Are you using lab files or TextGrid files for input? TextGrid files
2. Dictionary
   • Are you using a dictionary from MFA? If so, which one? No
   • If it's a custom dictionary, what is the phoneset? Very similar to the french_mfa model, but with a few additions and subtractions.
3. Acoustic model
   • If you're using an acoustic model, is it one download through MFA? If so, which one? NA
   • If it's a model you've trained, what data was it trained on? NA

Log file Please attach the log file for the run that encountered an error (by default these will be stored in ~/Documents/MFA). pg_log_global.txt

Desktop (please complete the following information):

• OS: Windows
• Version: 11 21H2
• Any other details about the setup (Cloud, Docker, etc): Python version 3.10.10, running in a conda environment

Additional context Here is the error message:

Exception ignored in atexit callback: <bound method ExitHooks.history_save_handler of <montreal_forced_aligner.command_line.mfa.ExitHooks object at 0x000001CC22E0A8C0>>
Traceback (most recent call last):
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\montreal_forced_aligner\command_line\mfa.py", line 102, in history_save_handler
    raise self.exception
  File "C:\Users\massi\anaconda3\envs\mfa\Scripts\mfa-script.py", line 9, in <module>
    sys.exit(mfa_cli())
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\rich_click\rich_group.py", line 21, in main
    rv = super().main(*args, standalone_mode=False, **kwargs)
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\montreal_forced_aligner\command_line\train_acoustic_model.py", line 111, in train_acoustic_model_cli
    trainer.train()
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\montreal_forced_aligner\acoustic_modeling\trainer.py", line 529, in train
    trainer.train()
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\montreal_forced_aligner\acoustic_modeling\base.py", line 488, in train
    self.initialize_training()
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\montreal_forced_aligner\acoustic_modeling\base.py", line 239, in initialize_training
    self._trainer_initialization()
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\montreal_forced_aligner\acoustic_modeling\monophone.py", line 329, in _trainer_initialization
    feat_dim = self.worker.get_feat_dim()
  File "C:\Users\massi\anaconda3\envs\mfa\lib\site-packages\montreal_forced_aligner\corpus\acoustic_corpus.py", line 917, in get_feat_dim
    job = self.jobs[0]
IndexError: list index out of range

I'm attempting to train a new acoustic model with the full phoneset I need by using an existing (pre-aligned) corpus in order to align additional data of the same type: every utterance therefore contains a single word. This error also occurs when multiprocessing is used (which is what I originally trued), regardless of the number of jobs specified. (The --single_speaker and --no_textgrid_cleanup flags were used since different errors resulted on other runs which did not include them.)

[mmcauliffe]

What were the errors when --single_speaker and --no_textrgrid_cleanup were omitted? I would recommend for something like this definitely not specifying --single_speaker, as it disables speaker adaptation.

[ppisljar]

i am getting the same, without using --single_speaker and --no_textgrid_cleanup

[diegotg2000]

I'm having the exact same error when trying to validate the data. I used mfa validate --clean my_corpus/ my_dictionary.txt, got this error and tried with --no_use_mp and --num_jobs 1 without success.

[ppisljar]

this works ok with my data/config on 2.0.6

[mmcauliffe]

Can you try upgrading to the latest version (2.2.13 or 2.2.14) and trying the command with --clean and see if that completes? I'm having trouble reproducing the error locally. Also try running with --no_use_postgres and see if that makes a difference?

[starmoon-1134]

I use the version of 2.2.14, and with --clean --no_use_postgres options, get the same error

[starmoon-1134]

I use the version of 2.2.14, and with --clean --no_use_postgres options, get the same error

My issue is due to missing the .lab file.

[electricazimuth]

I'm having this issue too. I've tried latest stable (2.2.15 & 2.2.17) and latest v3 (3.0.0a8). I believe the issue is that in the annotations (textgrid files) there are annotated phones that are silent and/or very short. No features can be extracted from them. The error seems to stem from the "get_feat_dim" call and results in an IndexError: list index out of range. I'm attaching the console output (no log file gets written) mfa-index-error.txt

[electricazimuth]

UPDATE: I've attempted running adapt on my corpus locally using multiple different conda installs and they all break. Attempting exactly the same process in a google notebook runs successfully (same commands, setup and corpus). Points to either an issue with some files locally, or some system library getting in the way. I have cleared out the MFA folder in the home directory and retried - errors still persist.

Are there other directories / files that need to be removed to create a fresh clean environment for MFA? Any other throughts on possible libraries I should check (each install is in conda so I was hoping that would prevent this type of issue)?