[BUG] can't align using acoustic model from 'mfa adapt'

[echodroff]

Debugging checklist

[x] Have you updated to latest MFA version? [x] Have you tried rerunning the command with the --clean flag?

Describe the issue I'm using the mfa adapt command to adapt the english_mfa acoustic model to some non-English speech. mfa adapt works fine, but then when I try to run mfa align with the new acoustic model, the alignment fails.

The alignment fails on adapted acoustic models that were trained on the latest version of MFA, as well as adapted acoustic models that used to do alignment just fine in January 2023 (trained using almost the exact same corpus/set-up).

For Reproducing your issue Please fill out the following:

1. Corpus structure
   • What language is the corpus in? Evenki but phone set adapted to English MFA
   • How many files/speakers? 5 files, 5 speakers
   • Are you using lab files or TextGrid files for input? TextGrids
2. Dictionary
   • Are you using a dictionary from MFA? If so, which one? no
   • If it's a custom dictionary, what is the phoneset? english_mfa
3. Acoustic model
   • If you're using an acoustic model, is it one download through MFA? If so, which one? adapted english_mfa
   • If it's a model you've trained, what data was it trained on? trained on Evenki data with the english_mfa phone set

Log file Please attach the log file for the run that encountered an error (by default these will be stored in ~/Documents/MFA). align_input.log

It looks suspicious that after each phone is the word "null"

When I use the original (unadapted) english_mfa model but same custom dictionary in mfa align, things work fine. Unfortunately, it doesn't generate the final log file for a comparison, though.

Desktop (please complete the following information):

• OS: [e.g. Windows, OSX, Linux] OSX
• Version [e.g. MacOSX 10.15, Ubuntu 20.04, Windows 10, etc] MacOSX Ventura 13.3.1, M1 chip
• Any other details about the setup (Cloud, Docker, etc)

Additional context Add any other context about the problem here.

[mmcauliffe]

Can you attach the adapted model or send it via email? Not sure what's going wrong there.

The nulls after phones are just a result of representing sets in yaml (yaml only has support for dictionary-types, and sets are similar to dictionary keys), so that shouldn't be related.

[echodroff]

Thanks for your quick response! The two models, relevant dictionaries, and files I'm trying to align are attached. The January23 version was trained with Evenki data and the pronDictIntermediateEvenkiEnglish.txt dictionary. The May23 version was trained with the exact same Evenki data but the pronDictIntermediateEvenkiEnglish_spn.txt dictionary (mostly treating foreign words as spn).

mfa align --clean ~/Desktop/train_full_adapted_spn/align_input pronDictIntermediateEvenkiEnglish evenki_adapted_english_jan23 ~/Desktop/train_full_adapted_spn/align_output

mfa align --clean ~/Desktop/train_full_adapted_spn/align_input pronDictIntermediateEvenkiEnglish_spn evenki_adapted_english_jan23 ~/Desktop/train_full_adapted_spn/align_output

mfa align --clean ~/Desktop/train_full_adapted_spn/align_input pronDictIntermediateEvenkiEnglish evenki_adapted_english_may23 ~/Desktop/train_full_adapted_spn/align_output

mfa align --clean ~/Desktop/train_full_adapted_spn/align_input pronDictIntermediateEvenkiEnglish_spn evenki_adapted_english_may23 ~/Desktop/train_full_adapted_spn/align_output

None of these lines work at the moment, but back in January, line 1 did work just fine.

align_input.zip https://drive.google.com/file/d/1ZdwBQzw9NsZ8APnaz4gibYMblZhDH8RH/view?usp=drive_web evenki_adapted_english_jan23.zip https://drive.google.com/file/d/1ylDXh7IATlU-SzNkxUU8FGeLHG2-HMx2/view?usp=drive_web evenki_adapted_english_may23.zip

https://drive.google.com/file/d/15GAtcK5Ke8LkPFEc1tXvWFFTDD3WTjku/view?usp=drive_web

pronDictIntermediateEvenkiEnglish_spn.txt pronDictIntermediateEvenkiEnglish.txt

[mmcauliffe]

Can you try adding the flag mfa align ... --position_dependent_phones false to the calls with the adapted models? It looks like the adapt command might not be collecting that flag from the original model.

[echodroff]

Amazing, that fixed it. Thank you so much!

[mmcauliffe]

One thing also to note is that using the default punctuation, word break markers and bracket settings is going to cause the foreign words to always be recognized with spn, so I'd recommend using --config_path evenki.yaml with the evenki.yaml as:

brackets: []
punctuation:
word_break_markers: " "

(assuming the text is already clean and doesn't need any punctuation stripped)

[echodroff]

Ah, definitely good to know. Long story, but in this case, I think we wanted to keep the foreign words as spn (even though at some point we did give them the rough transcription)

[echodroff]

Oh, I now see what you mean. Noted!

[echodroff]

Hm, I'm running into a similar issue with another Evenki-adapted English model, and adding --position_dependent_phones false no longer seems to be doing the trick. :/

mfa version 2.2.11

The model is here (base model is english_mfa 2.0): https://drive.google.com/file/d/1o55zvIn3kbUQK4jKPFFSOsIZcs44YfLK/view?usp=sharing

Dictionary is the same as above

The code: mfa align --clean /Users/eleanor/evenki/mfainput/short/tst pronDictIntermediateEvenkiEnglish evenki_adapted_english_short /Users/eleanor/evenki/train_test/short/train_full_adapted/tstoutput --position_dependent_phones false

It stops at the alignment stage.

The log file: compile_train_graphs.3.log

[echodroff]

Re-adapted and re-aligned with version 2.2.12 and also getting issues.

The Evenki-adapted model from version 12 is here (base model is english_mfa 2.0): https://drive.google.com/file/d/1KonW50GYgFGCtNfd8goT_z7OYKJuWiYd/view?usp=sharing

The log files: align.3.log

compile_train_graphs.3.log