NervanaSystems / deepspeech

DeepSpeech neon implementation
Apache License 2.0
222 stars 69 forks source link

train.py always uses gpu backend #35

Closed DawnStone closed 7 years ago

DawnStone commented 7 years ago

When trying to execute the Deepspeech train.py script with the -b cpu option, the gen_backend is executed for nervanagpu in error:

+ train_cmd='python train.py --manifest train:/mnt/scratch/jenkins/librispeech/train-clean-100/train-manifest.csv --manifest val:/mnt/scratch/jenkins/librispeech/dev-clean/val-manifest.csv -e 2 -z 32 -s /mnt/scratch/jenkins/librispeech/model_output.pkl -b cpu'
+ eval python train.py --manifest train:/mnt/scratch/jenkins/librispeech/train-clean-100/train-manifest.csv --manifest val:/mnt/scratch/jenkins/librispeech/dev-clean/val-manifest.csv -e 2 -z 32 -s /mnt/scratch/jenkins/librispeech/model_output.pkl -b cpu
++ python train.py --manifest train:/mnt/scratch/jenkins/librispeech/train-clean-100/train-manifest.csv --manifest val:/mnt/scratch/jenkins/librispeech/dev-clean/val-manifest.csv -e 2 -z 32 -s /mnt/scratch/jenkins/librispeech/model_output.pkl -b cpu
Traceback (most recent call last):
  File "train.py", line 24, in <module>
    from neon.backends import gen_backend
  File "/home/jenkins/workspace/Speech_Test_CPU/neon/backends/__init__.py", line 31, in <module>
    from neon.backends import nervanagpu
  File "/home/jenkins/workspace/Speech_Test_CPU/neon/backends/nervanagpu.py", line 40, in <module>
    from scikits.cuda import cublas
  File "/home/jenkins/workspace/Speech_Test_CPU/.venv2/local/lib/python2.7/site-packages/skcuda/cublas.py", line 22, in <module>
    from . import cuda
  File "/home/jenkins/workspace/Speech_Test_CPU/.venv2/local/lib/python2.7/site-packages/skcuda/cuda.py", line 9, in <module>
    from .cudart import *
  File "/home/jenkins/workspace/Speech_Test_CPU/.venv2/local/lib/python2.7/site-packages/skcuda/cudart.py", line 40, in <module>
    raise OSError('CUDA runtime library not found')
OSError: CUDA runtime library not found
DawnStone commented 7 years ago

This may be more appropriate as a neon issue.

DawnStone commented 7 years ago

This appears to only be a problem trying to execute the example with a non-gpu backend on a server that has GPU resources with neon.

Continued investigation will be for neon example execution environments.