search

brotchie / torch-ubuntu-gpu-ec2-install

Guide for installing Torch machine learning library onto a GPU EC2 instance running Ubuntu
72 stars 8 forks source link

Recurrent Neural Network Test Is Very Slow #3

Open frmsaul opened 8 years ago

frmsaul commented 8 years ago

Each batch takes roughly 3 seconds.

30 times slower than it should.

http://imgur.com/jpSyZQN

Operating system info:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS"

Is there anything I could do to speed it up?

brotchie commented 8 years ago

Hi Mate,

Looks like torch can't find your CUDA installation.

Does the deviceQuery sample utility run correctly?

cd ~
cuda-install-samples-7.0.sh .
cd NVIDIA_CUDA-7.0_Samples/1_Utilities/deviceQuery/
make
./deviceQuery
frisch1 commented 7 years ago

Hi Brotchie...

Sorry to be a bother. I've got this on a p2.xlarge with your AMI. Going along step by step. At this step: ./deviceQuery

I get this:

ubuntu@ip:~/NVIDIA_CUDA-7.0_Samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

modprobe: ERROR: ../libkmod/libkmod-module.c:809 kmod_module_insert_module() could not find module by name='nvidia_346'
modprobe: ERROR: could not insert 'nvidia_346': Function not implemented
cudaGetDeviceCount returned 38
-> no CUDA-capable device is detected
Result = FAIL

When I do a diagnostic, I get:

ubuntu@ip:~/NVIDIA_CUDA-7.0_Samples/1_Utilities/deviceQuery$ dpkg -l | grep -i nvidia
ii  bbswitch-dkms                             0.7-2ubuntu1                                        amd64        Interface for toggling the power on nVidia Optimus video cards
ii  libcuda1-346                              346.46-0ubuntu1                                     amd64        NVIDIA CUDA runtime library
ii  nvidia-346                                346.46-0ubuntu1                                     amd64        NVIDIA binary driver - version 346.46
ii  nvidia-346-dev                            346.46-0ubuntu1                                     amd64        NVIDIA binary Xorg driver development files
ii  nvidia-346-uvm                            346.46-0ubuntu1                                     amd64        NVIDIA Unified Memory kernel module
ii  nvidia-modprobe                           346.46-0ubuntu1                                     amd64        Load the NVIDIA kernel driver and create device files
ii  nvidia-opencl-icd-346                     346.46-0ubuntu1                                     amd64        NVIDIA OpenCL ICD
ii  nvidia-prime                              0.6.2                                               amd64        Tools to enable NVIDIA's Prime
ii  nvidia-settings                           346.46-0ubuntu1                                     amd64        Tool for configuring the NVIDIA graphics driver

Not sure where to go from here. Google failing me majorly. Any assistance would be appreciated.