naibaf7 / caffe

Caffe: a fast open framework for deep learning. With OpenCL and CUDA support.
http://caffe.berkeleyvision.org/
Other
85 stars 20 forks source link

Cannot use GPU in CPU-only Caffe: check mode with LibDNN and CLBlast on RX 480 #66

Closed itsnarsi closed 7 years ago

itsnarsi commented 7 years ago

Whenever I use Greantea and ViennaCL, my gpu crashes and shows error code 146. I saw in https://github.com/BVLC/caffe/issues/5285, @nucleargod solved the problem by installing only with LibDNN and CLBlast.

I compiled successfully with LibDNN and CLBlast without Greantea and ViennaCL.

However, caffe is going into cpu model only and giving following whenever I tried to check for gpu:

error:F0510 17:27:52.381860 12788 common.cpp:271] Cannot use GPU in CPU-only Caffe: check mode. Check failure stack trace: @ 0x7f168dbda5cd google::LogMessage::Fail() @ 0x7f168dbdc433 google::LogMessage::SendToLog() @ 0x7f168dbda15b google::LogMessage::Flush() @ 0x7f168dbdce1e google::LogMessageFatal::~LogMessageFatal() @ 0x7f168e2a3430 caffe::Caffe::EnumerateDevices() @ 0x408c4c device_query() @ 0x4082ce main @ 0x7f168cd14830 __libc_start_main @ 0x408b59 _start @ (nil) (unknown) Aborted (core dumped)

Here is my CMakeCache.txt.

Your system configuration

Operating system:UBUNTU 16.04 w/ AMD RX 480 BLAS: CLBLAST and openblas Python or MATLAB version (for pycaffe and matcaffe respectively):2.7

naibaf7 commented 7 years ago

No, that's not what https://github.com/BVLC/caffe/issues/5285 implies. You should build with these options:

If you use CPU-Only, then yes, no GPUs. Greentea, ViennaCL, LibDNN and clBLAS (not clBLAST) need to be present for the AMD GPU to work optimally.

itsnarsi commented 7 years ago

It Works!!! Thanks for the reply.

I have couple of questions I would like to ask you: 1) Is there a specific reason for not totally supporting CLBlast yet? 2) I've following your work with clCaffe, Greentea and Libdnn for a while now (Thanks for you involvement and investment in opencl for deep learning) and I would like to know are you anytime planning to support frameworks like TF or pytorch? 3) Do you know any good AMD GPU monitor in Linux, that would be really useful.

I would like to help. I not good in C++ (I hack through code to get my job done but that's it.) but I'm capable of handling myself in python and matlab. If there are any ongoing project I can try my best to help.

naibaf7 commented 7 years ago
  1. Not up to me, ask @CNugteren. It basically is fully supported by Caffe, but some AMD drivers apparently don't play along with all kernel configurations. CLBlas is by AMD, so the chances it works are higher. However there will be a new BLAS by AMD coming to Caffe in the near future as well.

  2. I'm not personally getting involved in TF or Torch, however TF is getting OpenCL support (slowly, but it will get there some day... note that auditing and complexity are greater with TF, more people are involved and approval by the maintainer takes time) and maybe LibDNN will be added to it, if useful (almost drop-in replacement for cuDNN, even though it's a bit slower).

Here at Caffe I've been fortunate enough to be the OpenCL branch maintainer, so I can develop new fun features more rapidly and bring them to the users quickly. Unfortunately I break things now and then, but I try my best to avoid bugs... it's just due to the more complex nature of the OpenCL branch (it supports about 8 BLAS libraries in total, has an infinite amount of "OpenCL compatible devices" to support, etc.).

  1. No, unfortunately there's not much managing tools for AMD GPUs.
itsnarsi commented 7 years ago

Thanks for the reply @naibaf7.

Hopefully AMD's MIopen brings gap between cross-platform comparability for different frameworks.

For now, I'm closing the issue.

naibaf7 commented 7 years ago

@itsnarsi MIopen and ROCm is a two-sided sword. On one hand it simplifies porting from CUDA to OpenCL, on the other hand I feel there is a big gap between AMD and many developers, as well as Intel and other companies preferring continuing on OpenCL. So it remains to hope that AMD keeps their whole MIopen and ROCm stack compatible to OpenCL, and as it looks like, they'll continue supporting Caffe OpenCL & Vega updates should come until the end of July. Stay tuned.