error when following instructions in data preparation (Librispeech)

[ghost]

The following fails:

$ th MakeLMDB.lua -rootPath prepare_datasets/libri_dataset -lmdbPath prepare_datasets/libri_lmdb -windowSize 0.02 -stride 0.01 -sampleRate 16000 -processes 8
luajit: MakeLMDB.lua:62: attempt to index field 'file' (a nil value)
stack traceback:
        MakeLMDB.lua:62: in function 'code'
        MakeLMDB.lua:170: in function 'f'
        (command line):4: in main chunk
        [C]: at 0x00405d50
MakeLMDB.lua:127: attempt to index local 'vec' (a nil value)
<parallel#000>  closing session

because at some point it tries:

$ find -L prepare_datasets/libri_dataset/train -type f -name '*.sph'

and the file extension in libri_dataset is .flac not .sph. Adding the extension parameter solves the problem:

$ th MakeLMDB.lua -rootPath prepare_datasets/libri_dataset  -lmdbPath prepare_datasets/libri_lmdb -windowSize 0.02 -stride 0.01 -sampleRate 16000 -processes 8 -audioExtension flac

I also suggest changing the "-processes 8" to "-processes $(nproc)"

[ghost]

Do you have similar training/testing scripts (like AN4CTCTrain.lua/AN4CTCTest.lua) for Librispeech? I guess I can use the same script with different parameters but finding the correct parameters is the problem.

$ th AN4CTCTrain.lua -trainingSetLMDBPath ./prepare_datasets/libri_lmdb/train/ -modelTrainingPath ./models/libri/
.../luajit: ./Loader.lua:30: batchSize larger than lmdb size!
stack traceback:
        [C]: in function 'assert'
        ./Loader.lua:30: in function '__init'
        .../ML/torch/install/share/lua/5.1/torch/init.lua:91: in function <.../torch/install/share/lua/5.1/torch/init.lua:87>
        [C]: in function 'indexer'
        ./Network.lua:61: in function 'init'
        AN4CTCTrain.lua:40: in main chunk
        [C]: in function 'dofile'
        .../torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00405d50

[SeanNaren]

Thanks so much! Done a few changes on the main branch using your suggestions. Usually if that's the case then the LMDB dataset wasn't found correctly or prepared correctly. I'll try to investigate further.

[ghost]

@SeanNaren: Could you respond to my question, please?

"Do you have similar training/testing scripts (like AN4CTCTrain.lua/AN4CTCTest.lua) for Librispeech?"

[SeanNaren]

I've modified the main branch removing the AN4 convention. These scripts should be able to take an LMDB database in the correct format regardless if it's AN4. However you will need to update the parameters like below on run:

th Train.lua -trainingSetLMDBPath /path/to/lmdb/train/ -validationSetLMDBPath /path/to/lmdb/test/

Hopefully this helps.