Evaluating pretraineed deepspeech model but getting error

[saurabhvyas]

I was following along the official documentation from neon github and neondeepspeech along with modelzoo documentation , I created a csv file and ran the following commands ( I am using ubuntu 16.04)

saurabhorange@orangepc:~/deepspeech_neon$ python evaluate.py --manifest val:test-clean/train-manifest.csv  --model_file weight.prm
python: can't open file 'evaluate.py': [Errno 2] No such file or directory
saurabhorange@orangepc:~/deepspeech_neon$ python speech/evaluate.py --manifest val:test-clean/train-manifest.csv  --model_file weight.prm
Traceback (most recent call last):
  File "speech/evaluate.py", line 22, in <module>
    from neon.backends import gen_backend
ImportError: No module named neon.backends

[tyler-nervana]

Are you running this from within the neon virtualenv?

[saurabhvyas]

No I am not using virtualenv OKay , due to some problems I reinstalled ubuntu and installed deepspeech and neon again (this time with virtualenv ) but I am still getting an error

(.venv2) saurabh@saurabh-Inspiron-5559:~/deepspeech/speech$ sudo python evaluate.py --manifest val:/home/saurabh/Downloads/train.csv  --model_file /home/saurabh/Downloads/train.prm
Traceback (most recent call last):
  File "evaluate.py", line 18, in <module>
    import numpy as np
ImportError: No module named numpy

But I do have numpy installed , I verified by typing pip install numpy

If I drop the word sudo from above command I get a different error

python evaluate.py --manifest val:/home/saurabh/Downloads/train.csv  --model_file /home/saurabh/Downloads/train.prm
2017-03-18 14:54:32,214 - neon.backends - WARNING - deterministic_update and deterministic args are deprecated in favor of specifying random seed
decode_thread_pool exception: Could not find file: "/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0000.flac"
decode_thread_pool exception: Could not find file: "/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0008.flac"
decode_thread_pool exception: Could not find file: "/home/saurabh/Downloads/LibriSpeech/1221/135767/1221-135767-0000.flac"
decode_thread_pool exception: Could not find file: "/home/saurabh/Downloads/LibriSpeech/1221/135767/1221-135767-0008.flac"
Segmentation fault (core dumped)

[tyler-nervana]

The scripts in this repo all have to be run from within the neon virtual environment, so you shouldn't be running anything with sudo (sudo temporarily brings you out into your root user's environment). The second error is more telling. Can you verify that the files at "/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0000.flac" do indeed exist? If they do, could you show me the first couple of lines from your train.csv file?

[saurabhvyas]

they do exist but just one directory up , so instead of '/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0000.flac , they exist at /home/saurabh/Downloads/LibriSpeech/test-clean/1221/135766/1221-135766-0000.flac

train.csv file looks like this

/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0000.flac,/home/saurabh/Downloads/LibriSpeech/transcripts/1221-135766-0000.txt
/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0001.flac,/home/saurabh/Downloads/LibriSpeech/transcripts/1221-135766-0001.txt
/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0002.flac,/home/saurabh/Downloads/LibriSpeech/transcripts/1221-135766-0002.txt

[saurabhvyas]

I have tried the same thing on a new machine with tensorflow gpu installled , and still getting an error ( even though this one seems different )

(.venv2) saurabh@saurabh-MS-7923:~/deepspeech_neon$ python speech/evaluate.py --manifest val:/home/saurabh/Downloads/LibriSpeech/test-clean/test.csv --model_file /home/saurabh/Downloads/test.prm
Traceback (most recent call last):
  File "speech/evaluate.py", line 47, in <module>
    args = parser.parse_args()
  File "/home/saurabh/neon/neon/util/argparser.py", line 357, in parse_args
    compat_mode=args.compat_mode)
  File "/home/saurabh/neon/neon/backends/__init__.py", line 108, in gen_backend
    deterministic=deterministic)
  File "/home/saurabh/neon/neon/backends/backend.py", line 468, in allocate_backend
    return Backend.backends[name](**kargs)
  File "/home/saurabh/neon/neon/backends/nervanagpu.py", line 761, in __init__
    deterministic=deterministic)
  File "/home/saurabh/neon/neon/backends/backend.py", line 480, in __init__
    self.rng = self.gen_rng(rng_seed)
  File "/home/saurabh/neon/neon/backends/nervanagpu.py", line 914, in gen_rng
    self.pcg = rng_mrg()
  File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/curandom.py", line 788, in __init__
    mrg32k3a_skip_ahead_sequence_source+random_skip_ahead64_source)
  File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/curandom.py", line 605, in __init__
    state_type, vector_type, generator_bits, additional_source)
  File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/curandom.py", line 440, in __init__
    self.module = module = pycuda.compiler.SourceModule(source, no_extern_c=True)
  File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/compiler.py", line 265, in __init__
    arch, code, cache_dir, include_dirs)
  File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/compiler.py", line 255, in compile
    return compile_plain(source, options, keep, nvcc, cache_dir, target)
  File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/compiler.py", line 78, in compile_plain
    checksum.update(preprocess_source(source, options, nvcc).encode("utf-8"))
  File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/compiler.py", line 50, in preprocess_source
    result, stdout, stderr = call_capture_output(cmdline, error_on_nonzero=False)
  File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pytools/prefork.py", line 223, in call_capture_output
    return forker.call_capture_output(cmdline, cwd, error_on_nonzero)
  File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pytools/prefork.py", line 59, in call_capture_output
    % (" ".join(cmdline), e))
pytools.prefork.ExecError: error invoking 'nvcc --preprocess -arch sm_52 -I/home/saurabh/neon/.venv2/lib/python2.7/site-packages/pycuda/cuda /tmp/tmpiv07oA.cu --compiler-options -P': [Errno 2] No such file or directory

[saurabhvyas]

Okay , I removed the leading ' / ' in path and that error got fixed ( on non gpu system , gpu system still has above error ) but now I am getting another error ( on cpu only system ) related to permission , I find it weird becuase I have manually verified and the .prm file has permissions for both read and write Here is the error message

2017-03-23 10:40:43,683 - neon.util.argparser - ERROR - No read access for model file home/saurabh/train.prm

[tyler-nervana]

they do exist but just one directory up

Could you post the command you ran to ingest the data? It's strange that you would get the wrong path out in such a way.

pytools.prefork.ExecError: error invoking 'nvcc --preprocess -arch sm_52 -I/home/saurabh/neon/.venv2/lib/python2.7/site-packages/pycuda/cuda /tmp/tmpiv07oA.cu --compiler-options -P': [Errno 2] No such file or directory

It seems the temporary file /tmp/tmpiv07oA.cu was not written out. Can you verify that you've set up all of your cuda environment variables (described here)

Okay , I removed the leading ' / ' in path and that error got fixed

This is just an earlier error. The '/' is required unless you are using relative paths.

[saurabhvyas]

Maybe I didnt add one of the environment variables , I copy pasted the environment variable statements from above mentioned link , and I retried to evaluate , I still get an error

python speech/evaluate.py --manifest val:/home/saurabh/Downloads/LibriSpeech/test-clean/test.csv --model_file /home/saurabh/Downloads/test.prm
2017-03-24 12:32:03,639 - neon.backends - WARNING - deterministic_update and deterministic args are deprecated in favor of specifying random seed
Traceback (most recent call last):
  File "speech/evaluate.py", line 103, in <module>
    use_wer=args.use_wer)
  File "/home/saurabh/deepspeech_neon/speech/utils.py", line 54, in get_wer
    model.initialize(dataset)
  File "/home/saurabh/neon/neon/models/model.py", line 128, in initialize
    self.layers.allocate()
  File "/home/saurabh/neon/neon/layers/container.py", line 295, in allocate
    l.allocate()
  File "/home/saurabh/neon/neon/layers/recurrent.py", line 1466, in allocate
    self.h_ff_buffer = self.be.zeros_like(self.outputs)
  File "/home/saurabh/neon/neon/backends/nervanagpu.py", line 1432, in zeros_like
    rounding=self.round_mode)._assign(0)
  File "/home/saurabh/neon/neon/backends/nervanagpu.py", line 146, in __init__
    self.gpudata = allocator(self.nbytes)
pycuda._driver.MemoryError: cuMemAlloc failed: out of memory

[tyler-nervana]

Which GPU are you using? You may be using a graphics card that doesn't have enough memory to run the full model, as deepspeech 2 is very large. Take a look at #10 for details on memory usage.

[saurabhvyas]

OKay , Ill check that post , I am using Nividia gtx 970 4gb

[saurabhvyas]

I think #10 helped , I reduced batch size parameter in evaluate.py ( to 4 ) and it worked! I guess I am gonna close it for now