Hi! I set up the docker container as instructed and have been able to use the demo.py successfully when i run it on cpu mode. However, when I try to run it on GPU, it fails after network initialization with:
I0823 12:38:11.485162 809 net.cpp:283] Network initialization done. I0823 12:38:11.746541 809 net.cpp:761] Ignoring source layer data I0823 12:38:11.746583 809 net.cpp:761] Ignoring source layer data_data_0_split I0823 12:38:11.783159 809 net.cpp:761] Ignoring source layer mbox_loss F0823 12:38:12.004873 809 math_functions.cu:26] Check failed: status == CUBLAS_STATUS_SUCCESS (13 vs. 0) CUBLAS_STATUS_EXECUTION_FAILED *** Check failure stack trace: *** Aborted (core dumped)
-I have tried adding CUDA_VISIBLE_DEVICES=0 python examples/text/demo.py, but it did not help.
-nvidia-smi shows the gpus as normal.
-I have been able to use nvidia-docker with other project on gpus succesfully, before, so i don't think this issue is with docker or nvidia-docker setup.
My host setup:
4xTesla P100-16GB
Ubuntu 16.04
Cuda 9.0
CuDNN 7.0.5
Hi! I set up the docker container as instructed and have been able to use the demo.py successfully when i run it on cpu mode. However, when I try to run it on GPU, it fails after network initialization with:
I0823 12:38:11.485162 809 net.cpp:283] Network initialization done. I0823 12:38:11.746541 809 net.cpp:761] Ignoring source layer data I0823 12:38:11.746583 809 net.cpp:761] Ignoring source layer data_data_0_split I0823 12:38:11.783159 809 net.cpp:761] Ignoring source layer mbox_loss F0823 12:38:12.004873 809 math_functions.cu:26] Check failed: status == CUBLAS_STATUS_SUCCESS (13 vs. 0) CUBLAS_STATUS_EXECUTION_FAILED *** Check failure stack trace: *** Aborted (core dumped)-I have tried adding CUDA_VISIBLE_DEVICES=0 python examples/text/demo.py, but it did not help. -nvidia-smi shows the gpus as normal. -I have been able to use nvidia-docker with other project on gpus succesfully, before, so i don't think this issue is with docker or nvidia-docker setup.
My host setup: 4xTesla P100-16GB Ubuntu 16.04 Cuda 9.0 CuDNN 7.0.5
I would be very thankful for any help with this!