search

cmusphinx / sphinxtrain

Acoustic model trainer for CMU Sphinx
Other
181 stars 112 forks source link

training is not properly restartable #37

Open dhdaines opened 2 years ago

dhdaines commented 2 years ago

If running sphinxtrain train and it fails (as it does, because the Perl scripts have bugs, but also as it might if using spot instances on $CLOUD) then there's no easy way to restart training.

Back in the old days we would just sit in our offices at CMU all night running scripts_pl/NN.step/s***ve_confg.pl manually, but I would prefer to be in the forest picking mushrooms these days.

This isn't rocket science, at the very least it could just restart from the step and iteration, though of course, it would be much better to rerun just the parts that failed. We're not even using GPUs so there's no issue with repeatability when doing that.

nshmyrev commented 2 years ago

There is -f option of sphinxtrain to run from any stage

https://github.com/cmusphinx/sphinxtrain/blob/master/scripts/sphinxtrain#L164

not well documented though

dhdaines commented 2 years ago

Yes, I've been using that, though it restarts the stage from the beginning, while it should restart from an iteration (or ideally just rerun the job that failed and carry on).

One might question the utility of improving things like this in SphinxTrain, but it's still a considerably nicer training tool than the monstrosity that is the Kaldi scripts :)