search

amplab / SparkNet

Distributed Neural Networks for Spark
MIT License
604 stars 172 forks source link

is current ubuntu library GPU cudnn enabled? #128

Open nhe150 opened 8 years ago

nhe150 commented 8 years ago

I have tried to run with GoogleNet(more computing). The forwardbackward took really long time. Is the .so library provided is GPU cudnn enabled? There is a new rc3-1.2 of javacpp caffe preset release. What is the procedure to adopt rc3-1.2 release? Included is the logs::

workerId = 0 getWeights took 0.05 s transformInto took 1.942 s ForwardBackward took 99.302 s

transformInto took 2.67 s ForwardBackward took 109.891 s

transformInto took 1.676 s ForwardBackward took 122.732 s

transformInto took 1.571 s ForwardBackward took 131.627 s

transformInto took 1.463 s ForwardBackward took 138.765 s

transformInto took 1.564 s ForwardBackward took 140.662 s

transformInto took 1.66 s ForwardBackward took 141.29 s

nhe150 commented 8 years ago

Above is for batch size 256. Here is logs for batch size 128:: transformInto took 1.157 s ForwardBackward took 45.425 s

transformInto took 0.982 s ForwardBackward took 46.187 s

transformInto took 0.861 s ForwardBackward took 50.866 s

transformInto took 0.868 s ForwardBackward took 53.46 s

transformInto took 0.89 s ForwardBackward took 56.189 s

robertnishihara commented 8 years ago

Hmm that's strange. Which app are you running (and are you using the current master)? Are you using our AMI? What kind of cluster are you on?

If you run watch -n 0 nvidia-smi while the app is running, do you see any GPU utilization?

nhe150 commented 8 years ago

I am running local hadoop cluster. I saw gpu when issue nvidia-smi command. But the forwardbackward is too slow...