Slower performance in baumwelch 0.3.6 compared to 0.3.1

[mmcauliffe]

(this is not a Pynini issue, so if there's a better place to put this, let me know)

An MFA user is noticing some poor performance training G2P models in the latest version (https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner/issues/289). From my own tests, generating alignments using baumwelch==0.3.6 is slower about 3x slower in the course of training a G2P model:

• 67,284 words as training data
• 8 jobs
• 0.3.1 alignment time: ~10 minutes
• 0.3.6 alignment time : ~30 minutes

I noticed in the NEWS that there was a change in 0.3.2 for stepwise training with a learning rate parameter. I'm assuming this results in better alignments, but is it expected to take longer? For speed ups, would the same 50 iterations be necessary in the newer version, and are there any other parameters I can tweak to improve performance?

[kylebgorman]

Turn up the batch size with --batch_size (if 0 is passed, each batch is the full FAR, the old behavior), reduce the number of iterations with --max_iters, and/or turn up the learning rate with --lr.

I can't say I've experimented enough to know how to quickly tune these but for your case I'd try mid-sized batches of maybe 100 or 200 and I bet you'll converge in just a few epochs. 50 is a lot of epochs for this kind of problem.

[mmcauliffe]

Ah perfect, thanks, I'll play around with those parameters more and find a good place for them, looks like pretty good performance just setting --batch_size to 200.