SeanNaren / deepspeech.torch

Speech Recognition using DeepSpeech2 network and the CTC activation function.
MIT License
259 stars 73 forks source link

error when following instructions in data preparation (Librispeech) #53

Closed ghost closed 8 years ago

ghost commented 8 years ago

The following fails:

$ th MakeLMDB.lua -rootPath prepare_datasets/libri_dataset -lmdbPath prepare_datasets/libri_lmdb -windowSize 0.02 -stride 0.01 -sampleRate 16000 -processes 8
luajit: MakeLMDB.lua:62: attempt to index field 'file' (a nil value)
stack traceback:
        MakeLMDB.lua:62: in function 'code'
        MakeLMDB.lua:170: in function 'f'
        (command line):4: in main chunk
        [C]: at 0x00405d50
MakeLMDB.lua:127: attempt to index local 'vec' (a nil value)
<parallel#000>  closing session

because at some point it tries:

$ find -L prepare_datasets/libri_dataset/train -type f -name '*.sph'

and the file extension in libri_dataset is .flac not .sph. Adding the extension parameter solves the problem:

$ th MakeLMDB.lua -rootPath prepare_datasets/libri_dataset  -lmdbPath prepare_datasets/libri_lmdb -windowSize 0.02 -stride 0.01 -sampleRate 16000 -processes 8 -audioExtension flac

I also suggest changing the "-processes 8" to "-processes $(nproc)"

ghost commented 8 years ago

Do you have similar training/testing scripts (like AN4CTCTrain.lua/AN4CTCTest.lua) for Librispeech? I guess I can use the same script with different parameters but finding the correct parameters is the problem.

$ th AN4CTCTrain.lua -trainingSetLMDBPath ./prepare_datasets/libri_lmdb/train/ -modelTrainingPath ./models/libri/
.../luajit: ./Loader.lua:30: batchSize larger than lmdb size!
stack traceback:
        [C]: in function 'assert'
        ./Loader.lua:30: in function '__init'
        .../ML/torch/install/share/lua/5.1/torch/init.lua:91: in function <.../torch/install/share/lua/5.1/torch/init.lua:87>
        [C]: in function 'indexer'
        ./Network.lua:61: in function 'init'
        AN4CTCTrain.lua:40: in main chunk
        [C]: in function 'dofile'
        .../torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00405d50
SeanNaren commented 8 years ago

Thanks so much! Done a few changes on the main branch using your suggestions. Usually if that's the case then the LMDB dataset wasn't found correctly or prepared correctly. I'll try to investigate further.

ghost commented 8 years ago

@SeanNaren: Could you respond to my question, please?

"Do you have similar training/testing scripts (like AN4CTCTrain.lua/AN4CTCTest.lua) for Librispeech?"

SeanNaren commented 8 years ago

I've modified the main branch removing the AN4 convention. These scripts should be able to take an LMDB database in the correct format regardless if it's AN4. However you will need to update the parameters like below on run:

th Train.lua -trainingSetLMDBPath /path/to/lmdb/train/ -validationSetLMDBPath /path/to/lmdb/test/

Hopefully this helps.