Closed baruchih closed 4 years ago
I did the same command as you pointed, everything works for me.
The problem here could be that driver is not appropriate for the CUDA version we use inside docker. Doesn't matter which CUDA you installed on your machine, only CUDA driver matters. The error you have is pointing exactly to this Error initializing CUDA runtime. Check your CUDA device is visible to the OS and you have installed the correct driver. Try running the nvidia-smi utility to debug any driver issues.
Could you confirm that for other tools/frameworks you don't have problems of CUDA driver with your CUDA 10.1 and 10.2 installed?
Mine machine shows Driver Version: 418.116.00 CUDA Version: 10.1
. Probably your driver is too new for the version we build. One possible solution for you (in case of not downgrade your driver) is to rebuild base docker image for the flashlight and then rebuild base docker for the wav2letter.
Thanks for your reply @tlikhomanenko
I tried to rebuild the docker, and the issue still persisted.
Than I tried on a different environment. It passed with no issue.
I just had to expose the GPU using export CUDA_VISIBLE_DEVICES=1
I will re-install the driver on my first environment, and try again.
Thanks for the help!
Hello, I have the same problem with the Docker image, following the wiki : sudo docker run --gpus all --rm -itd --ipc=host --name w2l wav2letter/wav2letter:cuda-latest (by the way runtime=nvidia does not work with the new nvidia docker -> wiki correction needed)
I installed docker, Nvidia docker and driver as required.
Thanks for any help!
@iggygeek
what cuda driver version do you have? Also could you try to build image on your machine and test if it works?
My apologies, it was just that all GPUs where busy on this machine ...
@baruchih feel free to reopen if the problem still persists.
Bug Description
Hello,
I have downloaded the latest image of wav2letter-cuda (Dockerhub
DIGEST:sha256:228eb912d5a61a151de4abf9cd9c25a4aa3a9a912ed2d538c7a3a574d948b11a
)When running the
make test
within the docker the tests fails. This has been reproduced on 2 separate computers, with cuda 10.1, and cuda 10.2. Same test fails.Test result:
When running the
W2lCommonTest
for example the output is:Any ideas on what's the issue here?
Reproduction Steps
docker pull wav2letter/wav2letter:cuda-latest
docker run --gpus all --rm -itd --ipc=host --name w2l wav2letter/wav2letter:cuda-latest
/root/wav2letter/build
and runmake test
Platform and Hardware
Ubuntu 18.04
nvidia-smi
output: