search

mravanelli / pytorch-kaldi

pytorch-kaldi is a project for developing state-of-the-art DNN/RNN hybrid speech recognition systems. The DNN part is managed by pytorch, while feature extraction, label computation, and decoding are performed with the kaldi toolkit.
2.37k stars 446 forks source link

Can we resume training from the epoch we got interruption #241

Open sun-peach opened 4 years ago

sun-peach commented 4 years ago

Hi, my computer's power was down and the experiment was interrupted at epoch 13. Can we resume the training from where it was interrupted?

Thank you.

TParcollet commented 4 years ago

Well, yes. The model is stored each epoch. So if the training stops at epoch 13 (let's say at 50%) then it should restart at epoch 13 0%

sun-peach commented 3 years ago

Thank you. But do we need to set some flag to do that or pytorch kaldi will do it automatically? @TParcollet

mravanelli commented 3 years ago

My suggestion is to move to SpeechBrain though: https://speechbrain.github.io/ SpeechBrain is a new project which is much richer, easy-to-use, well-documented than the old pytorch-kaldi one.

On Tue, 16 Mar 2021 at 01:38, sun-peach @.***> wrote:

Thank you. But do we need to set some flag to do that or pytorch kaldi will do it automatically? @TParcollet https://github.com/TParcollet

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mravanelli/pytorch-kaldi/issues/241#issuecomment-799966868, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEA2ZVWN2SHTXHOKYQLNYKTTD3VEHANCNFSM4PPJOSLQ .

sun-peach commented 3 years ago

@mravanelli I have already noted that. I am doing research on that. Thank you.