search

naibaf7 / caffe

Caffe: a fast open framework for deep learning. With OpenCL and CUDA support.
http://caffe.berkeleyvision.org/
Other
86 stars 20 forks source link

ld: symbol(s) not found for architecture x86_64 #7

Closed ColemanGariety closed 9 years ago

ColemanGariety commented 9 years ago

OS: OS X 10.9 MacBook Pro Retina 2013 Greentea, Viennacl-BLAS CUDA 7.029 GCC 4.8.4

I'm able to build the main caffe properly but not this fork. I installed gcc48 and set it in config in addition to linking the ViennaCL headers.

Here's what I get when I make all -j8: http://pastebin.com/QQNaMkz4

naibaf7 commented 9 years ago

@paulbroyles @lavania You guys use OS X, right? Any ideas?

From my perspective it just seems that your OpenCL, OpenCV and Google Flags is not linked correctly (not found), but their headers have been found. Try to find versions of those libraries that are binary compatible with x86_64 GCC, I guess... And there are also issues showing up that relate to faulty/missing/wrong libstdc++11

ColemanGariety commented 9 years ago

@naibaf7 ah so they just don't work with g++ 4.8.4?

How would I go about installing double versions and using those only within this caffe install (since I don't use 4.8.4 all the time).

naibaf7 commented 9 years ago

@JacksonGariety No ideas, not an OS X expert. Let's see if @paulbroyles and @lavania can help you since they got it running on their OS X machines. Maybe they have some hints for you.

ColemanGariety commented 9 years ago

@naibaf7 thanks in advance!

lavania commented 9 years ago

@JacksonGariety

I actually used clang++ to build the system.

If you want to use g++ then my suggestion would be to try something like :

set CXX=/path/to/bin/g++; export CXX;

Note that this is if you are using a bash shell.

Once you do this, then caffe might be able to pick up g++ if you use cmake. I think the Makefile explicitly tells the system to use clang++ (Makefile sets the CXX in this case) if OS X is found to be the operating system. You can try changing it to g++ in the Makefile and try that route (using make).

ColemanGariety commented 9 years ago

@lavania I'm not wedded to g++, the project just lists at as a "new optional requirement."

When I use clang or g++ (other than 48) I get this: LD -o .build_release/lib/libcaffe.so ld: library not found for -lgomp

Using g++-48 solves the problem, but then I get the error mentioned above.

paulbroyles commented 9 years ago

@JacksonGariety: I also built using clang++. I experimented with g++, and I think I had a similar problem at one point, but I'm afraid I don't remember what came of it. If I can work it out I'll follow up.

The clang error you're getting is because clang doesn't support OpenMP. You can resolve it by editing the Makefile and removing the -fopenmp argument from CXXFLAGS and LINKFLAGS. (Of course, I presume in doing so you lose parallelism... But I'm very much a novice in these areas.)

ColemanGariety commented 9 years ago

@paulbroyles @lavania I get the same error, or at least a very similar one, when I use clang++. It has roughly the same number of "symbols not found."

What version of clang are you using? And did you use cmake?

ColemanGariety commented 9 years ago

@naibaf7 I'm thinking using cmake may solve my problem, but when I do the following:

  1. mkdir build; cd build
  2. cmake ..
  3. make all

nothing happens, same if I cd .. and make all. make seems to break if I run cmake on my caffe dir. This doesn't happen with the main project. Am I using cmake wrong?

paulbroyles commented 9 years ago

@JacksonGariety Apple LLVM version 6.1.0 (clang-602.0.53).

I'm getting the undefined symbols error at the same point you are using g++ 4.8, but it's building fine with clang.

I will add that I have not had any luck with cmake--it dies on me very quickly--but following the steps you listed DOES cause it to start building for me.

ColemanGariety commented 9 years ago

@paulbroyles you're running 10.10 then? I'm running 10.9 and LLVM 6.0 perhaps that's why.

Edit: the fact that your cmake works surprises me since it doesn't seem to work no matter what I do and that's should be out-of-box right? Am I cloning the right repo? https://github.com/naibaf7/caffe.git

Edit 2: nevermind, recloned the repo and cmake started working.

ColemanGariety commented 9 years ago

@naibaf7 when using cmake I get:

nvcc fatal   : redefinition of argument 'std'

Even though I have 'use CUDA' commented out and Greentea set to 1.

jyegerlehner commented 9 years ago

@JacksonGariety

If you are getting nvcc errors when building via cmake, I expect you must not have switched off the CUDA dependency in CMakeLists.txt. You would probably need to set these options as follows:

caffe_option(USE_GREENTEA "Build Caffe with OpenCL support" ON)

caffe_option(USE_CUDNN "Build Caffe with cuDNN libary support" OFF)

caffe_option(USE_CUDA "Build Caffe with CUDA support" OFF)

And re-run cmake. I have seen cases where I needed to delete the build directory entirely, then re-create it, and run cmake again in an empty directory. make clean isn't always enough.

[Edit] Oops, I see you said you're using MacBook Pro Retina, which I see has Nvidia GPU. So my comments above may not apply. I had assumed above that you were looking to run on an OpenCL machine.

lavania commented 9 years ago

@JacksonGariety

The 'symbols not found' (with opencv) are there if your opencv was not built with libstdc++ . The Makefile enforces the use of libstdc++ for OSX. I think that commenting out these lines (I think in the latest Makefile they are 322 and 323) might do the trick.

I didn't use cmake to build the system, so not sure about the issues there..

naibaf7 commented 9 years ago

Thanks @lavania and @paulbroyles and @jyegerlehner Could anyone of you write the appropriate conditionals and flags into the makefile and/or CMake so that it works more easily on OS X without having to mess around every time? You could then just make a pull request. As you already have working build infrastructures of this project it should be easy to do.

That would be really great. If I were to do it myself, it would be very messy because I can never test it out myself (no OS X present). I currently do testing on Fedora 22 and Ubuntu 14.04, so that's how the build infrastructure is set up.

bhack commented 9 years ago

@naibaf7 I don't know if some OSX users it is still interested in https://github.com/BVLC/caffe/pull/1218. This could generally help OSX test build on the whole caffe.

bhack commented 9 years ago

Seems that OSX build it is enabled again on Travis. See https://github.com/travis-ci/travis-ci/issues/216#issuecomment-124138138

ColemanGariety commented 9 years ago

@naibaf7 So I got caffe to build on my (osx) system by doing the following:

  1. comment out CXXFLAGS and LINKFLAGS in the fork's Makefile (lines 322 and 323)
  2. switch OFF CUDA in CMakeLists.txt
  3. use cmake to build

But then I got this error: terminated by signal SIGSEGV (Address boundary error) Which is a segfault on my shell.

So I did what made sense: deugged it. Naturally, it was segfaulting at import caffe so I looked at the report and tracked down this ticket: which suggested the system python was interfering, which was correct. So I got down to one python version, linked it, rebuilt this caffe fork, then moved the caffe/python directory to my user folder.

But now an even more daunting problem. I simply get: SHITTTTTTTTTTTTTT You're running CPU man =D Not sure where to troubleshoot from here. I got caffe built successfully and it imports properly but it still thinks I want to use the CPU!

EDIT: The above steps get caffe to build for me on OS X though so I hope this is helpful! Can't get it to work with DeepDreamVideo though, but that's a separate issue.

naibaf7 commented 9 years ago

@JacksonGariety We're getting there ;D.... and oh well, OS X does seem like an utter mess!!! Now: What command did you run? What does

 ./build/tools/caffe device_query

give you?

Does this run, also:

./build/tools/caffe time -model models/bvlc_alexnet/deploy.prototxt -gpu=0

On another note: Why do you want to use OpenCL when you got nVidia/CUDA? Something wrong with that? :)

ColemanGariety commented 9 years ago

@naibaf7 My rMPB has an Intel GPU :cry:

device_query spits out:

Total devices: 2
CUDA devices: 0
OpenCL devices: 2
Device id:                     0
Device backend:                OpenCL
Backend details:               Apple: OpenCL 1.2 (Jul 29 2014 21:24:39)
Device vendor:                 Intel
Name:                          Intel(R) Core(TM) i7-4750HQ CPU @ 2.00GHz
Total global memory:           8589934592
Device id:                     1
Device backend:                OpenCL
Backend details:               Apple: OpenCL 1.2 (Jul 29 2014 21:24:39)
Device vendor:                 Intel
Name:                          Iris Pro
Total global memory:           1610612736

And yes that command seems to run 50 iterations in 26731 ms which is way better than CPU. If only I could get deep dream to use that.

Maybe this is just a problem with DeepDreamVideo: https://github.com/graphific/DeepDreamVideo

EDIT: I forgot a flag on my DeepDreamVideo command (oops) but anyway I'm back to terminated by signal SIGSEGV (Address boundary error)

naibaf7 commented 9 years ago

Oooh... In this case, try: ./build/tools/caffe time -model models/bvlc_alexnet/deploy.prototxt -gpu=1 because ./build/tools/caffe time -model models/bvlc_alexnet/deploy.prototxt -gpu=0 will use your CPU on OpenCL ;) (which is faster than CPU...)

You should also be able to set the GPU to 1 on DeepDreamVideo.

Please report performance of the Iris Pro, I never have seen one of those!

ColemanGariety commented 9 years ago

@naibaf7 that one works as well with no segfault (see my edit?):

I forgot a flag on my DeepDreamVideo command (oops) but anyway I'm back to terminated by signal SIGSEGV (Address boundary error)
naibaf7 commented 9 years ago

Try first if simple stuff like ./build/tools/caffe time -model models/bvlc_alexnet/deploy.prototxt -gpu=0 and ./build/tools/caffe time -model models/bvlc_alexnet/deploy.prototxt -gpu=1 works before jumping to 3rd party libraries. Please report the last 5 to 7 lines of output you get by using those commands.

ColemanGariety commented 9 years ago

@naibaf7 my mistake.

But this thread was helpful! The steps I outlined in this comment got me working on OS X.

Basically there's something wrong with the env variables on lines 322 and 323 of the Makefile that prevents it from building on OS X.

I opened graphific/DeepDreamVideo#51 to troubleshoot the segfault which seems to be isolated to DeepDreamVideo.

naibaf7 commented 9 years ago

@JacksonGariety Ok, I hope you can figure it out. I would still like to have your performance report on the two commands above, if you don't mind. I would like to add them to the performance overview in this fork. Especially the Iris Pro is interesting.

Thanks.

ColemanGariety commented 9 years ago

@naibaf7 here you go:

gpu = 0

gpu = 1

naibaf7 commented 9 years ago

@JacksonGariety Thanks! That is simply great! So it is definitely working correctly on both devices. However you may have noticed your CPU is quite a bit faster than your GPU. So when using DeepDreamVideo... essentially you're better off using the CPU. You could however use both the CPU and GPU interlocked on half the frames you want to process each. That way you'd get about a 1.7x speedup on the whole task.

ColemanGariety commented 9 years ago

@naibaf7 that's funny since two days ago I finished rendering the video with CPU and wanted to do it again faster with the GPU. Good to know that GPU isn't always faster with caffe!

ColemanGariety commented 9 years ago

@naibaf7 hmmm it looks like the segfault is directly from caffe. not surprising.

Here's the pastebin if you wanna take a look: http://pastebin.com/3uXY5hcx It's really odd. Segfault is pretty serious. Maybe ViennaCL just has some problems with my GPU.

naibaf7 commented 9 years ago

@JacksonGariety Yes there are a few things to keep in mind with that. It's also not specifically optimized for the Intel GPUs so far. And a lot also depends on the BLAS, which is ViennaCL-BLAS here it seems. It mostly depends on how fast your device can do matrix-matrix multiplications. With CPUs on OpenCL, I still use the system CPU BLAS that you have installed. That is heavily optimized for your Haswell CPU.

With that segfault, there's actually two things you can try:

#ifndef GREENTEA_QUEUE_COUNT
#define GREENTEA_QUEUE_COUNT 8
#endif

to:

#ifndef GREENTEA_QUEUE_COUNT
#define GREENTEA_QUEUE_COUNT 1
#endif

Then recompile the library. I have a feeling 8 queues is a bit too much on that network and thus it might crash. It has to be something like that because the alexnet test commands are sound.

ColemanGariety commented 9 years ago

@naibaf7 thanks for the tips! You can close this now since it's a separate issue.

bhack commented 9 years ago

@naibaf7 A recent article on Intel https://software.intel.com/en-us/articles/sgemm-for-intel-processor-graphics

ColemanGariety commented 9 years ago

@naibaf7 lowering the queue count did not work. Oh well, I have my video. Maybe the DDV owner will respond. Thanks for the help again!

naibaf7 commented 9 years ago

@JacksonGariety https://github.com/naibaf7/caffe/commit/2806b26006f66a92916427c35cd2710fcad08a6f Should fix the remaining issues. Just pull and recompile. Seems like the OpenCL queues were not an issue after all. You can set them back up to 4 or 8 if it works.

naibaf7 commented 9 years ago

@bhack Thanks. Yes there is still some work to do with GEMM/GEMV. Still in contact with the clBLAS developers as well as ViennaCL. But my main focus currently is device abstraction and multi-GPU training as they already have with CUDA.