Closed saurabhvyas closed 7 years ago
Are you running this from within the neon virtualenv?
No I am not using virtualenv OKay , due to some problems I reinstalled ubuntu and installed deepspeech and neon again (this time with virtualenv ) but I am still getting an error
(.venv2) saurabh@saurabh-Inspiron-5559:~/deepspeech/speech$ sudo python evaluate.py --manifest val:/home/saurabh/Downloads/train.csv --model_file /home/saurabh/Downloads/train.prm
Traceback (most recent call last):
File "evaluate.py", line 18, in <module>
import numpy as np
ImportError: No module named numpy
But I do have numpy installed , I verified by typing pip install numpy
If I drop the word sudo from above command I get a different error
python evaluate.py --manifest val:/home/saurabh/Downloads/train.csv --model_file /home/saurabh/Downloads/train.prm
2017-03-18 14:54:32,214 - neon.backends - WARNING - deterministic_update and deterministic args are deprecated in favor of specifying random seed
decode_thread_pool exception: Could not find file: "/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0000.flac"
decode_thread_pool exception: Could not find file: "/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0008.flac"
decode_thread_pool exception: Could not find file: "/home/saurabh/Downloads/LibriSpeech/1221/135767/1221-135767-0000.flac"
decode_thread_pool exception: Could not find file: "/home/saurabh/Downloads/LibriSpeech/1221/135767/1221-135767-0008.flac"
Segmentation fault (core dumped)
The scripts in this repo all have to be run from within the neon virtual environment, so you shouldn't be running anything with sudo (sudo temporarily brings you out into your root user's environment). The second error is more telling. Can you verify that the files at "/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0000.flac"
do indeed exist? If they do, could you show me the first couple of lines from your train.csv
file?
they do exist but just one directory up , so instead of '/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0000.flac , they exist at /home/saurabh/Downloads/LibriSpeech/test-clean/1221/135766/1221-135766-0000.flac
train.csv file looks like this
/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0000.flac,/home/saurabh/Downloads/LibriSpeech/transcripts/1221-135766-0000.txt
/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0001.flac,/home/saurabh/Downloads/LibriSpeech/transcripts/1221-135766-0001.txt
/home/saurabh/Downloads/LibriSpeech/1221/135766/1221-135766-0002.flac,/home/saurabh/Downloads/LibriSpeech/transcripts/1221-135766-0002.txt
I have tried the same thing on a new machine with tensorflow gpu installled , and still getting an error ( even though this one seems different )
(.venv2) saurabh@saurabh-MS-7923:~/deepspeech_neon$ python speech/evaluate.py --manifest val:/home/saurabh/Downloads/LibriSpeech/test-clean/test.csv --model_file /home/saurabh/Downloads/test.prm
Traceback (most recent call last):
File "speech/evaluate.py", line 47, in <module>
args = parser.parse_args()
File "/home/saurabh/neon/neon/util/argparser.py", line 357, in parse_args
compat_mode=args.compat_mode)
File "/home/saurabh/neon/neon/backends/__init__.py", line 108, in gen_backend
deterministic=deterministic)
File "/home/saurabh/neon/neon/backends/backend.py", line 468, in allocate_backend
return Backend.backends[name](**kargs)
File "/home/saurabh/neon/neon/backends/nervanagpu.py", line 761, in __init__
deterministic=deterministic)
File "/home/saurabh/neon/neon/backends/backend.py", line 480, in __init__
self.rng = self.gen_rng(rng_seed)
File "/home/saurabh/neon/neon/backends/nervanagpu.py", line 914, in gen_rng
self.pcg = rng_mrg()
File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/curandom.py", line 788, in __init__
mrg32k3a_skip_ahead_sequence_source+random_skip_ahead64_source)
File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/curandom.py", line 605, in __init__
state_type, vector_type, generator_bits, additional_source)
File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/curandom.py", line 440, in __init__
self.module = module = pycuda.compiler.SourceModule(source, no_extern_c=True)
File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/compiler.py", line 265, in __init__
arch, code, cache_dir, include_dirs)
File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/compiler.py", line 255, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/compiler.py", line 78, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode("utf-8"))
File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pycuda/compiler.py", line 50, in preprocess_source
result, stdout, stderr = call_capture_output(cmdline, error_on_nonzero=False)
File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pytools/prefork.py", line 223, in call_capture_output
return forker.call_capture_output(cmdline, cwd, error_on_nonzero)
File "/home/saurabh/neon/.venv2/local/lib/python2.7/site-packages/pytools/prefork.py", line 59, in call_capture_output
% (" ".join(cmdline), e))
pytools.prefork.ExecError: error invoking 'nvcc --preprocess -arch sm_52 -I/home/saurabh/neon/.venv2/lib/python2.7/site-packages/pycuda/cuda /tmp/tmpiv07oA.cu --compiler-options -P': [Errno 2] No such file or directory
Okay , I removed the leading ' / ' in path and that error got fixed ( on non gpu system , gpu system still has above error ) but now I am getting another error ( on cpu only system ) related to permission , I find it weird becuase I have manually verified and the .prm file has permissions for both read and write Here is the error message
2017-03-23 10:40:43,683 - neon.util.argparser - ERROR - No read access for model file home/saurabh/train.prm
they do exist but just one directory up
Could you post the command you ran to ingest the data? It's strange that you would get the wrong path out in such a way.
pytools.prefork.ExecError: error invoking 'nvcc --preprocess -arch sm_52 -I/home/saurabh/neon/.venv2/lib/python2.7/site-packages/pycuda/cuda /tmp/tmpiv07oA.cu --compiler-options -P': [Errno 2] No such file or directory
It seems the temporary file /tmp/tmpiv07oA.cu
was not written out. Can you verify that you've set up all of your cuda environment variables (described here)
Okay , I removed the leading ' / ' in path and that error got fixed
This is just an earlier error. The '/' is required unless you are using relative paths.
Maybe I didnt add one of the environment variables , I copy pasted the environment variable statements from above mentioned link , and I retried to evaluate , I still get an error
python speech/evaluate.py --manifest val:/home/saurabh/Downloads/LibriSpeech/test-clean/test.csv --model_file /home/saurabh/Downloads/test.prm
2017-03-24 12:32:03,639 - neon.backends - WARNING - deterministic_update and deterministic args are deprecated in favor of specifying random seed
Traceback (most recent call last):
File "speech/evaluate.py", line 103, in <module>
use_wer=args.use_wer)
File "/home/saurabh/deepspeech_neon/speech/utils.py", line 54, in get_wer
model.initialize(dataset)
File "/home/saurabh/neon/neon/models/model.py", line 128, in initialize
self.layers.allocate()
File "/home/saurabh/neon/neon/layers/container.py", line 295, in allocate
l.allocate()
File "/home/saurabh/neon/neon/layers/recurrent.py", line 1466, in allocate
self.h_ff_buffer = self.be.zeros_like(self.outputs)
File "/home/saurabh/neon/neon/backends/nervanagpu.py", line 1432, in zeros_like
rounding=self.round_mode)._assign(0)
File "/home/saurabh/neon/neon/backends/nervanagpu.py", line 146, in __init__
self.gpudata = allocator(self.nbytes)
pycuda._driver.MemoryError: cuMemAlloc failed: out of memory
Which GPU are you using? You may be using a graphics card that doesn't have enough memory to run the full model, as deepspeech 2 is very large. Take a look at #10 for details on memory usage.
OKay , Ill check that post , I am using Nividia gtx 970 4gb
I think #10 helped , I reduced batch size parameter in evaluate.py ( to 4 ) and it worked! I guess I am gonna close it for now
I was following along the official documentation from neon github and neondeepspeech along with modelzoo documentation , I created a csv file and ran the following commands ( I am using ubuntu 16.04)