search

hughperkins / DeepCL

OpenCL library to train deep convolutional neural networks
Mozilla Public License 2.0
867 stars 199 forks source link

Mac OSX install #32

Closed outlace closed 8 years ago

outlace commented 8 years ago

How do you get this to work on Mac OSX?

hughperkins commented 8 years ago

Hi, sorry for the delay in replying; I like to think I normally answer faster than that. So... building on Mac is a bit hit and miss unfortunately, it depends on a few factors, works for some people, less well for others. I think you should try using the linux method for building, and see what happens, ie:

git clone --recursive https://github.com/hughperkins/DeepCL.git
cd DeepCL
mkdir build
cd build
ccmake ..
# in ccmake:
# - press 'c'/configure
# - choose the options you want
# - press 'c' /configure again
# - press 'g' / generate, then `q` / quit
make -j 4 install
./deepcl_unittests

More details on pre-requisites and stuff at https://github.com/hughperkins/DeepCL/blob/master/doc/Build.md

Edit: by the way, please let me know how this goes, success/failure, and any logs and so on, if any issues.

outlace commented 8 years ago

Sorry for late reply, was out of town.

Tried installing. Seemed to install okay, but I tried ./deepcl_unittests and get this error

$ ./deepcl_unittests dyld: Library not loaded: libEasyCL.dylib Referenced from: /Users/brandon/DeepCL/build/./deepcl_unittests Reason: image not found Trace/BPT trap: 5

Although I've installed/built EasyCL

hughperkins commented 8 years ago

Hi outlace, Ok, I think you will need to set some environment variables, such as DYLD_LIBRARY_PATH, something like:

export DYLD_LIBRARY_PATH=../dist/lib
export LD_LIBRARY_PATH=../dist/lib
./deepcl_unittests
outlace commented 8 years ago

I think it worked? Does that one failed test matter?

[ OK ] testGpuOp.addscalarinplace (8 ms) [----------] 4 tests from testGpuOp (35 ms total)

[----------] 1 test from testjpeghelper [ RUN ] testjpeghelper.writeread [ OK ] testjpeghelper.writeread (13 ms) [----------] 1 test from testjpeghelper (13 ms total)

[----------] Global test environment tear-down [==========] 159 tests from 29 test cases ran. (67203 ms total) [ PASSED ] 158 tests. [ FAILED ] 1 test, listed below: [ FAILED ] testNorbLoader.load1000

1 FAILED TEST YOU HAVE 2 DISABLED TESTS

hughperkins commented 8 years ago

That one test doesnt matter. It means that the norb dataset is not present, which is fine. Looks like it works, you have succeeded :-)

outlace commented 8 years ago

Sweet! Thank you very much!!

outlace commented 8 years ago

Actually.. Now I'm trying to install the Python wrapper and I get this error:

Failed building wheel for DeepCL Failed to build DeepCL Installing collected packages: DeepCL Running setup.py install for DeepCL Complete output from command /Users/brandon/anaconda/bin/python3 -c "import setuptools, tokenize;file='/private/var/folders/zt/qjk01pyj4hn34zyhq9h50r1c0000gn/T/pip-build-k35n57y6/DeepCL/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /var/folders/zt/qjk01pyj4hn34zyhq9h50r1c0000gn/T/pip-sgk6frfv-record/install-record.txt --single-version-externally-managed --compile: version: 8.0.0 running install running build running build_ext building 'PyDeepCL' extension gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/brandon/anaconda/include -arch x86_64 -Imysrc -Imysrc/lua -I/Users/brandon/anaconda/include/python3.4m -c PyDeepCL.cxx -o build/temp.macosx-10.5-x86_64-3.4/PyDeepCL.o In file included from PyDeepCL.cxx:251: In file included from /Users/brandon/DeepCL/dist/include/deepcl/DeepCL.h:6: In file included from /Users/brandon/DeepCL/dist/include/easycl/EasyCL.h:31: In file included from /Users/brandon/DeepCL/dist/include/easycl/deviceinfo_helper.h:22: /Users/brandon/DeepCL/dist/include/easycl/mystdint.h:11:14: fatal error: 'cstdint' file not found

include 

             ^
1 error generated.
error: command 'gcc' failed with exit status 1
----------------------------------------

Command "/Users/brandon/anaconda/bin/python3 -c "import setuptools, tokenize;file='/private/var/folders/zt/qjk01pyj4hn34zyhq9h50r1c0000gn/T/pip-build-k35n57y6/DeepCL/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /var/folders/zt/qjk01pyj4hn34zyhq9h50r1c0000gn/T/pip-sgk6frfv-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/zt/qjk01pyj4hn34zyhq9h50r1c0000gn/T/pip-build-k35n57y6/DeepCL

GOFAI commented 8 years ago

Looking at the setup.py file, I think that the problem is that under Darwin clang isn't told to set the '-std=c++0x' option.

hughperkins commented 8 years ago

@GOFAI Ah, ok. Any thoughts on how to update the setup.py to include this option?

hughperkins commented 8 years ago

Ah, in this bit:

if osfamily == 'Windows':
    compile_options.append('/EHsc')
elif osfamily == 'Linux':
    compile_options.append('-std=c++0x')
    compile_options.append('-g')
else:
    pass

outface, dont suppose.. can you open a python prompt, and do:

import platform
print platform.uname()[0]
outlace commented 8 years ago

import platform print(platform.uname()[0]) Darwin

GOFAI commented 8 years ago

Don't have time to test at the moment, but I think it's probably as simple as:

if osfamily == 'Windows':
    compile_options.append('/EHsc')
elif osfamily == 'Linux' or osfamily == 'Darwin':
    compile_options.append('-std=c++0x')
    compile_options.append('-g')
else:
    pass
hughperkins commented 8 years ago

Ok. Fixed in cdb0438 outface, can you pull down the latest version, and retry?

m-smith commented 8 years ago

The above didn't work for me, but the following did! 

if osfamily == 'Windows':
    compile_options.append('/EHsc')
elif osfamily in ['Linux']:
    compile_options.append('-std=c++0x')
    compile_options.append('-g')
elif osfamily == 'Darwin':
    compile_options.append('-mmacosx-version-min=10.7')
    compile_options.append('-stdlib=libc++')
    compile_options.append('-g')
hughperkins commented 8 years ago

Thanks! Merged.

outlace commented 8 years ago

I'm getting a different error now $ python setup.py install ... ... running install_lib running build_ext building 'PyDeepCL' extension creating build creating build/temp.macosx-10.5-x86_64-3.5 gcc -fno-strict-aliasing -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/brandonbrown/anaconda/include -arch x86_64 -Imysrc -Imysrc/lua -I/Users/brandonbrown/anaconda/include/python3.5m -c PyDeepCL.cxx -o build/temp.macosx-10.5-x86_64-3.5/PyDeepCL.o -mmacosx-version-min=10.7 -stdlib=libc++ -g PyDeepCL.cxx:250:10: fatal error: 'CppRuntimeBoundary.h' file not found

include "CppRuntimeBoundary.h"

     ^

1 error generated. error: command 'gcc' failed with exit status 1

m-smith commented 8 years ago

is it possible that you forgot to run source dist/bin/activate.sh ?

outlace commented 8 years ago

Whoops, indeed I did. Thanks

On Wednesday, November 4, 2015, Matthew Smith notifications@github.com wrote:

is it possible that you forgot to run source dist/bin/activate.sh ?

— Reply to this email directly or view it on GitHub https://github.com/hughperkins/DeepCL/issues/32#issuecomment-153925310.

outlace commented 8 years ago

Now I have an error importing. Sorry guys...Thanks so much for your patience and help

Brandons-Air:DeepCL brandon$ python Python 3.5.0 |Continuum Analytics, Inc.| (default, Oct 20 2015, 14:39:26) [GCC 4.2.1 (Apple Inc. build 5577)] on darwin Type "help", "copyright", "credits" or "license" for more information.

import PyDeepCL Traceback (most recent call last): File "", line 1, in ImportError: dlopen(/Users/brandonbrown/anaconda/lib/python3.5/site-packages/DeepCL-0.0.0-py3.5-macosx-10.5-x86_64.egg/PyDeepCL.cpython-35m-darwin.so, 2): Library not loaded: libclBLAS.2.dylib Referenced from: /Users/brandonbrown/anaconda/lib/python3.5/site-packages/DeepCL-0.0.0-py3.5-macosx-10.5-x86_64.egg/PyDeepCL.cpython-35m-darwin.so Reason: image not found

hughperkins commented 8 years ago

Hmmmm, somehow missing the libclBLAS.2.dylib library in the classpath. It looks like the PyDeepCL library references it, but it's not being found.

There seem to be two possibilities I can think of:

The first possibility is easier to check, so let's check that first. Plesae can you do:

echo $DYLD_LIBRARY_PATH
ls /Users/brandon/DeepCL/dist/lib
outlace commented 8 years ago

Brandons-Air:DeepCL brandon$ echo $DYLD_LIBRARY_PATH

Brandons-Air:DeepCL brandon$ ls /Users/brandon/DeepCL/dist/lib cmake libclBLAS.2.4.0.dylib libclew.1.0.0.dylib libDeepCL.dylib libclBLAS.2.dylib libclew.dylib libEasyCL.dylib libclBLAS.dylib libdeepcl_gtest.a

hughperkins commented 8 years ago

hmmm, can you try setting your DYLD_LIBRARY_PATH, maybe something like:

export DYLD_LIBRARY_PATH=/Users/brandon/DeepCL/dist/lib

... and then retry using python

m-smith commented 8 years ago

For some reason, I needed to set the DYLD_FALLBACK_LIBRARY_PATH environment variable...same as above just 

export DYLD_FALLBACK_LIBRARY_PATH=/Users/brandon/DeepCL/dist/lib
outlace commented 8 years ago

That did it! You guys are amazing, thank you so much

Brandons-Air:DeepCL brandonbrown$ python Python 3.5.0 |Continuum Analytics, Inc.| (default, Oct 20 2015, 14:39:26) [GCC 4.2.1 (Apple Inc. build 5577)] on darwin Type "help", "copyright", "credits" or "license" for more information.

import PyDeepCL cl = PyDeepCL.DeepCL() Using Apple , OpenCL platform: Apple Using OpenCL device: HD Graphics 5000 initializing clblas

hughperkins commented 8 years ago

Ah, needed the DYLD_FALLBACK_LIBRARY_PATH? Cool. Nice! (sorry about my occasional repeated postings by the way, my network is a bit iffy :-P )

hughperkins commented 8 years ago

Added DYLD_LIBRARY_PATH and DYLD_FALLBACK_LIBRARY_PATH in 911f53b. I think I can close this now, right?

Dexdev08 commented 8 years ago

total n00b here... [ FAILED ] testsinglebatch.imagesize5_filtersize3_batchsize2_10filters [ FAILED ] testNorbLoader.load1000 [ FAILED ] testCopyLocal.basic

I have tried your instructions above... I am failing 2 more unit tests... testCopyLocal.basic sounds important...

hughperkins commented 8 years ago

Yes, the other two are ok to fail. norbloader fails, because it needs the norb dataset, which you dont have. The singlebatch test is a bit stochastic, and fails occasionally (I should fix that really).

testcopylocal doesnt normally fail.... in fact I dont remember it ever failing... if all the other tests are passing, it probably doesnt matter really though. I think you can probably just go ahead and use the library. However, if you do want to dig into why, I think you need to provide:

Dexdev08 commented 8 years ago

Thanks! I'm using OSX Mavericks on a Macbook Air (late 2013) with the intel HD 5000 card (opencl enabled)... There's a reason why i'm looking for opencl... as I have not exploited this feature that much.

I'm attaching clinfo here... not sure how to retrieve the output of testCopyLocal.basic (shame shame on me) clinfo.txt

hughperkins commented 8 years ago

Ok. Can you paste the full output of running ./deepcl_unittests tests=testCopyLocal.* please?

On my laptop, this gives me:

DeepCL/build$ ./deepcl_unittests tests=testCopyLocal.*
args: ./deepcl_unittests --gtest_filter=testCopyLocal.*
Note: Google Test filter = testCopyLocal.*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from testCopyLocal
[ RUN      ] testCopyLocal.basic
Using NVIDIA Corporation , OpenCL platform: NVIDIA CUDA
Using OpenCL device: GeForce 940M
0 0 0 0 
1 2 3 4 
5 6 7 8 
9 10 11 12 

0 0 0 0 
[       OK ] testCopyLocal.basic (809 ms)
[----------] 1 test from testCopyLocal (809 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (809 ms total)
[  PASSED  ] 1 test.
Dexdev08 commented 8 years ago

Hi Hugh,

Thanks for your patience. I've ran the test and dumped the error message into the text file below. error_msg.txt

Dexter

hughperkins commented 8 years ago

Ok... not quite sure how to fix that to be honest. Works ok on my machine. Let's at least get rid of some of those opencl build warnings (or try to...), and those two failing tests, which should be done (or at least, started) in commit 022b6a3 , on master branch

hughperkins commented 8 years ago

Perhaps you can try running the EasyCL tests first, since presumably it's fairly on something fairly basic (I guess....). Please can you try the following? :

git clone --recursive https://github.com/hughperkins/EasyCL.git
cd EasyCL
mkdir build
cd build
ccmake ..
# press 'c' for configure, set:
#  BUILD_TESTS=ON, 
#  CMAKE_BUILD_TYPE=Debug
# press 'c' for configure
# press 'g' for generate
# => ccmake should exit.  Then do:
make -j 4 install
cp ../test/*.cl .
./easycl_unittests

... and provide the results.

Dexdev08 commented 8 years ago

Hi! Sorry just came from work. I've ran this... is the warning after pressing g in ccmake relevant --> "Policy cmp0042 is not set: macosx_rpath is enabled by default ..." easycltests.txt

hughperkins commented 8 years ago

is the warning after pressing g in ccmake relevant --> "Policy cmp0042 is not set: macosx_rpath is enabled by default ..."

dont think so, because the problem is probably opencl-related. rpath is used to locate libraries, ie .sos and .dylibs. If those weren't being located the program wouldnt run at all, would crash immediately with a message like couldnt find libOpenCL.so, something like that.

easycltests.txt

Hmmm, those all pass...

Dexdev08 commented 8 years ago

i think this library is usable... i have tried running test_deepcl.py... and it seems to be running, showing a test accuracy of 95.62%

hughperkins commented 8 years ago

Ok, cool :-)

Dexdev08 commented 8 years ago

thanks!

Dexdev08 commented 8 years ago

Hi Hugh,

I think we can compile a readme for OSX installation.

I've compiled my experience here, and lifted some of your instructions.

http://mynotebookis.blogspot.sg/2015/11/installing-deepcl-on-macbook-air-2013.html

GOFAI commented 8 years ago

Might it be worthwhile to put together a formula for homebrew-science, or should that wait until the next official release?

hughperkins commented 8 years ago

@Dexdev08 Thats awesome! Thanks! :-)

hughperkins commented 8 years ago

@GOFAI That sounds cool. Maybe I will fire up Jenkins and compile the latest code, and we use that?

hughperkins commented 8 years ago

Ok. Builds working now. Created v8.1.2 release https://github.com/hughperkins/DeepCL/releases/tag/v8.1.2

GOFAI commented 8 years ago

I've made two Homebrew formulas. The first, head-only one worked fine with brew install -i --HEAD DeepCL: https://gist.github.com/GOFAI/6071778631ba3d7e8ebf

It would be better to use the tarball of the versioned release, but unfortunately it turns out that if one tries that the recursive submodules EasyCL and clMathLibraries don't get cloned: https://gist.github.com/GOFAI/6b07ecdbace369ddd4ca When I try to step through this one in interactive mode (brew install -i DeepCL), I hit the following snags in cmake:

Setting build type to 'RelWithDebInfo'

 CMake Error at
 /usr/local/Cellar/cmake/3.3.2/share/cmake/Modules/ExternalProject.cmake:1871
 (message):
   No download info given for 'EasyCL-external' and its source directory:

    /tmp/deepcl20151120-8228-1sxnmek/DeepCL-8.1.2/EasyCL

   is not an existing non-empty directory.  Please specify one of:

    * SOURCE_DIR with an existing non-empty directory
    * URL
    * GIT_REPOSITORY
    * HG_REPOSITORY
    * CVS_REPOSITORY and CVS_MODULE
    * SVN_REVISION
    * DOWNLOAD_COMMAND
Call Stack (most recent call first):

 /usr/local/Cellar/cmake/3.3.2/share/cmake/Modules/ExternalProject.cmake:2356
 (_ep_add_download_command)
   cmake/build_EasyCL.cmake:3 (ExternalProject_Add)
   CMakeLists.txt:66 (INCLUDE)

 CMake Error at
 /usr/local/Cellar/cmake/3.3.2/share/cmake/Modules/ExternalProject.cmake:1871
 (message):
   No download info given for 'clBLAS-external' and its source directory:

    /tmp/deepcl20151120-8228-1sxnmek/DeepCL-8.1.2/clMathLibraries/clBLAS/src

    is not an existing non-empty directory.  Please specify one of:

    * SOURCE_DIR with an existing non-empty directory
    * URL
    * GIT_REPOSITORY
    * HG_REPOSITORY
    * CVS_REPOSITORY and CVS_MODULE
    * SVN_REVISION
    * DOWNLOAD_COMMAND
 Call Stack (most recent call first):

 /usr/local/Cellar/cmake/3.3.2/share/cmake/Modules/ExternalProject.cmake:2356
 (_ep_add_download_command)
   cmake/build_clBLAS.cmake:9 (ExternalProject_Add)
   CMakeLists.txt:70 (INCLUDE)
hughperkins commented 8 years ago

Nice! :-)

It would be better to use the tarball of the versioned release, but unfortunately it turns out that if one tries that the recursive submodules EasyCL and clMathLibraries don't get cloned

Hmmmm...ok... well... couple of ideas, not sure how doable these are?

GOFAI commented 8 years ago

Some version of the second method is canonical for brew formulae. It would be nice to be able to build against other brew formulae such as clBLAS, but if need be there is apparently a way to pull in the subcomponents as versioned tar files. I just need to untar them into the right folders before running ccmake, correct?

hughperkins commented 8 years ago

Yes, you will need to untar as follows:

GOFAI commented 8 years ago

So I've got clBLAS, EasyCL, and clew staging correctly, but I'm not sure what arguments to pass to cmake, and using homebrew's default *std_cmake_args results in the following output:

JOHNNIAC:~ Walrus$ brew install DeepCL
==> Downloading https://github.com/hughperkins/DeepCL/archive/v8.1.2.tar.gz
Already downloaded: /Library/Caches/Homebrew/deepcl-8.1.2.tar.gz
==> Downloading https://github.com/hughperkins/EasyCL/archive/b9023cdf25c10524ad
Already downloaded: /Library/Caches/Homebrew/deepcl--EasyCL-2.zip
==> Downloading https://github.com/hughperkins/clBLAS/archive/bdaf1f9ef3d97168fd
Already downloaded: /Library/Caches/Homebrew/deepcl--clBLAS-171.zip
==> Downloading https://github.com/hughperkins/clew/archive/1d2752564b6af98d2412
Already downloaded: /Library/Caches/Homebrew/deepcl--clew-02.zip
==> mkdir build
==> cd build
==> cmake .. -DCMAKE_C_FLAGS_RELEASE= -DCMAKE_CXX_FLAGS_RELEASE= -DCMAKE_INSTALL
Last 15 lines from /Users/Walrus/Library/Logs/Homebrew/deepcl/03.cmake:
2015-11-21 12:13:36 -0800

cmake
..
-DCMAKE_C_FLAGS_RELEASE=
-DCMAKE_CXX_FLAGS_RELEASE=
-DCMAKE_INSTALL_PREFIX=/usr/local/Cellar/deepcl/8.1.2
-DCMAKE_BUILD_TYPE=Release
-DCMAKE_FIND_FRAMEWORK=LAST
-DCMAKE_VERBOSE_MAKEFILE=ON
-Wno-dev

CMake Error: The source directory "/tmp/deepcl20151121-14817-1dqypbk" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.

READ THIS: https://git.io/brew-troubleshooting
GOFAI commented 8 years ago

Sorted out the problem--builds successfully and ./deepcl_unittests passes 158 of 159 tests. I just need to clean it up to meet homebrew's submission standards, including some sort of nominal test.

One issue is that using deepcl_unittests as the required test will run afoul of Brew Test Bot, because the single failed test causes the whole thing to count as a failure:

JOHNNIAC:bin Walrus$ brew test DeepCL
Testing deepcl
==> /usr/local/Cellar/deepcl/8.1.2/bin/deepcl_unittests
Last 15 lines from /Users/Walrus/Library/Logs/Homebrew/deepcl/01.deepcl_unittests:

[----------] 1 test from testjpeghelper
[ RUN      ] testjpeghelper.writeread
[       OK ] testjpeghelper.writeread (1 ms)
[----------] 1 test from testjpeghelper (1 ms total)

[----------] Global test environment tear-down
[==========] 159 tests from 30 test cases ran. (59058 ms total)
[  PASSED  ] 158 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DATA_testNorbLoader.load1000

 1 FAILED TEST
  YOU HAVE 2 DISABLED TESTS

Error: deepcl: failed
hughperkins commented 8 years ago

Oh.... seems I havent quite excluded the DATA_* tests (of which there are one). I'll fix that.

PS Nice work Edward :-)