Closed dimatter closed 7 years ago
1.7.0 crashes like so:
(.venv2) dimatter@qb:~/neonSpeech/deepspeech/speech$ python train.py --manifest train:/home/dimatter/neonSpeech/deepspeech/librispeech/training.csv --manifest val:/home/dimatter/neonSpeech/deepspeech/librispeech/validation.csv -e 1 -z 10 -s ../new.prm --model_file ../librispeech_16_epochs.prm
2017-07-02 00:52:58,059 - neon.util.argparser - WARNING - No schedule given for model serialization, using default 1
2017-07-02 00:52:59,704 - neon.backends - WARNING - deterministic_update and deterministic args are deprecated in favor of specifying random seed
Traceback (most recent call last):
File "train.py", line 176, in <module>
nesterov=True)
TypeError: __init__() got an unexpected keyword argument 'nesterov'
1.8 & 1.9 crash with the assertion fail...
when trying to continue training the pre-trained model on CPU/mkl:
2017-07-02 05:57:39,616 - neon.models.model - WARNING - Problems restoring existing RNG state: algorithm must be 'MT19937'
starting a clean model on mkl:
[src/conv.c:151] err (-127)
[src/conv.c:152] err (-127)
[src/conv.c:153] err (-127)
[src/conv.c:172] err (-1)
[src/conv.c:173] err (-1)
[src/conv.c:174] err (-1)
[src/conv.c:175] err (-1)
[src/conv.c:176] err (-1)
[src/conv.c:177] err (-1)
[src/conv.c:178] err (-1)
[src/conv.c:179] err (-1)
[src/conv.c:180] err (-1)
[src/conv.c:204] err (-1)
wrong input for try_convert!
Segmentation fault (core dumped)
I figured out that it was my batch size that it was complaining about.
python train.py --manifest train:/home/dimatter/neonSpeech/deepspeech/librispeech/training.csv --manifest val:/home/dimatter/neonSpeech/deepspeech/librispeech/validation.csv -e 1 -z 16 -s ../new.prm --model_file ../librispeech_16_epochs.prm -b gpu
2017-07-02 06:49:00,051 - neon.util.argparser - WARNING - No schedule given for model serialization, using default 1
2017-07-02 06:49:01,669 - neon.backends - WARNING - deterministic_update and deterministic args are deprecated in favor of specifying random seed
and silently crashes.
The "silent crashing" on GPU is actually caused by the model finishing training. The epoch index is stored with the model file, so in order to do one more epoch of training, you have to use -e 17
.
We don't yet have support for Deepspeech2 on MKL, unfortunately.