oadams
commented
3 years ago Speaking mostly for the ESPnet recipes here, I think there's a good argument for not bothering with this feature. The argument is that the vast majority of the time is spent in the training step. The time for preprocessing beforehand is relatively small. I'm not totally against this feature or anything, but just don't think it brings as much to the table versus other things time could be spent on and this feature feels like a vector for bugs.
(My comment about outputting the directory where the model is and taking one in I meant with regards to importing/exporting fully trained models).
Currently there is no easy way to save where the user is up to in the training steps, or load saved state. This is inconvenient for end-users and developers.
This could be implemented simply (as @oadams suggested), by outputting the directory where the model is, and loading by feeding the directory back in.
It would be good to include easy-to-access info about the state of the saved model. Saved models that are complete could be loaded as pre-trained models. It may be useful to look at the way ESPnet handles the upload of trained models to Zenodo. See step 4.3 in the Colab.