Closed rickeycarter closed 7 years ago
It doesn't look like you called install_keras()
anywhere. That should install Keras in an r-tensorflow virtual environment which should be found by default by the keras and tensorflow R packages.
I did run install_keras() in a variety of ways, but apparently I didn't include those errors above. As a summary, I can have it such that the library(keras) will cause an error or that when keras is installed with install_keras(tensorflow="1.1.0-gpu") it can't find the links to the cuDNN libraries. I've tried running system commands to define the symbolic links or the environment variables. R seems to want to default back to the default mac python (/usr/bin/python). It seems to be related to environment variable since running "spyder" at the command prompt in a terminal shell will work with keras and tensorflow, but launching spyder from Anaconda misses the cuDNN libraries just like is happening within R.
I think the problem may be related to the fact that Mac desktop applications don't see what's in .bash_profile by default (you can probably confirm this by calling Sys.getenv()
from within the RStudio session). I think for this you need to create a .plist file for environment variables as described here: https://apple.stackexchange.com/questions/106355/setting-the-system-wide-path-environment-variable-in-mavericks
Here's a test of what's happening
library(keras)
install_keras(tensorflow="1.1.0-gpu")
now <- Sys.time()
# the data, shuffled and split between train and test sets
data <- dataset_mnist()
This is the log
R version 3.3.3 (2017-03-06) -- "Another Canoe"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin13.4.0 (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
[Workspace loaded from ~/.RData]
> library(keras)
> install_keras(tensorflow="1.1.0-gpu")
Using existing virtualenv at ~/.virtualenvs/r-tensorflow
Upgrading pip ...
Collecting pip
Using cached pip-9.0.1-py2.py3-none-any.whl
Installing collected packages: pip
Successfully installed pip-9.0.1
Upgrading wheel ...
Collecting wheel
Using cached wheel-0.30.0-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel-0.30.0
Upgrading setuptools ...
Collecting setuptools
Using cached setuptools-36.4.0-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-36.4.0
Installing TensorFlow ...
Collecting tensorflow-gpu==1.1.0
Using cached tensorflow_gpu-1.1.0-cp27-cp27m-macosx_10_11_x86_64.whl
Collecting h5py
Using cached h5py-2.7.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting pyyaml
Collecting requests
Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting Pillow
Using cached Pillow-4.2.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting keras
Using cached Keras-2.0.8-py2.py3-none-any.whl
Collecting scipy
Using cached scipy-0.19.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting werkzeug>=0.11.10 (from tensorflow-gpu==1.1.0)
Using cached Werkzeug-0.12.2-py2.py3-none-any.whl
Collecting six>=1.10.0 (from tensorflow-gpu==1.1.0)
Using cached six-1.10.0-py2.py3-none-any.whl
Collecting wheel (from tensorflow-gpu==1.1.0)
Using cached wheel-0.30.0-py2.py3-none-any.whl
Collecting numpy>=1.11.0 (from tensorflow-gpu==1.1.0)
Using cached numpy-1.13.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting mock>=2.0.0 (from tensorflow-gpu==1.1.0)
Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting protobuf>=3.2.0 (from tensorflow-gpu==1.1.0)
Using cached protobuf-3.4.0-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests)
Using cached idna-2.6-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests)
Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached certifi-2017.7.27.1-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting olefile (from Pillow)
Collecting funcsigs>=1; python_version < "3.3" (from mock>=2.0.0->tensorflow-gpu==1.1.0)
Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock>=2.0.0->tensorflow-gpu==1.1.0)
Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting setuptools (from protobuf>=3.2.0->tensorflow-gpu==1.1.0)
Using cached setuptools-36.4.0-py2.py3-none-any.whl
Installing collected packages: werkzeug, six, wheel, numpy, funcsigs, pbr, mock, setuptools, protobuf, tensorflow-gpu, h5py, pyyaml, idna, urllib3, certifi, chardet, requests, olefile, Pillow, scipy, keras
Successfully installed Pillow-4.2.1 certifi-2017.7.27.1 chardet-3.0.4 funcsigs-1.0.2 h5py-2.7.1 idna-2.6 keras-2.0.8 mock-2.0.0 numpy-1.13.1 olefile-0.44 pbr-3.1.1 protobuf-3.4.0 pyyaml-3.12 requests-2.18.4 scipy-0.19.1 setuptools-36.4.0 six-1.10.0 tensorflow-gpu-1.1.0 urllib3-1.22 werkzeug-0.12.2 wheel-0.30.0
Installation of TensorFlow complete.
Installation of Keras complete.
> data <- dataset_mnist()
Using TensorFlow backend.
Error in py_module_import(module, convert = convert) :
ImportError: Traceback (most recent call last):
File "/Users/<<username>>/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "/Users/<<username>>/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "/Users/<<username>>/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
ImportError: dlopen(/Users/<<username>>/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so, 10): Library not loaded: @rpath/libcublas.8.0.dylib
Referenced from: /Users/<<username>>/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/tensorflow/pyth
In addition: Warning message:
In stop(e, call. = FALSE) : additional arguments ignored in stop()
The results of Sys.getenv() is this
> Sys.getenv()
__CF_USER_TEXT_ENCODING
0x61AAF1E4:0x0:0x0
Apple_PubSub_Socket_Render
/private/tmp/com.apple.launchd.gJ21R1AQ9l/Render
COMMAND_MODE unix2003
DISPLAY /private/tmp/com.apple.launchd.ned6xG1YtT/org.macosforge.xquartz:0
DYLD_FALLBACK_LIBRARY_PATH
/Library/Frameworks/R.framework/Resources/lib:/Users/<<username>>/lib:/usr/local/lib:/usr/lib::
EDITOR vi
GIT_ASKPASS rpostback-askpass
HOME /Users/<<username>>
LANG en_US.UTF-8
LC_CTYPE en_US.UTF-8
LN_S ln -s
LOGNAME <<username>>
MAKE make
PAGER /usr/bin/less
PATH /Users/<<username>>/.virtualenvs/r-tensorflow/bin:/Users/<<username>>/.virtualenvs/r-tensorflow/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/usr/texbin
R_BROWSER /usr/bin/open
R_BZIPCMD /usr/bin/bzip2
R_DOC_DIR /Library/Frameworks/R.framework/Resources/doc
R_GZIPCMD /usr/bin/gzip
R_HOME /Library/Frameworks/R.framework/Resources
R_INCLUDE_DIR /Library/Frameworks/R.framework/Resources/include
R_LIBS_SITE
R_LIBS_USER ~/Library/R/3.3/library
R_PAPERSIZE a4
R_PDFVIEWER /usr/bin/open
R_PLATFORM x86_64-apple-darwin13.4.0
R_PRINTCMD lpr
R_QPDF /Library/Frameworks/R.framework/Resources/bin/qpdf
R_RD4PDF times,inconsolata,hyper
R_SESSION_TMPDIR /var/folders/w6/ng3s613x6d9gyvfnl5l2yknjhtnwg4/T//Rtmp10Jeq3
R_SHARE_DIR /Library/Frameworks/R.framework/Resources/share
R_SYSTEM_ABI osx,gcc,gxx,gfortran,?
R_TEXI2DVICMD /usr/local/bin/texi2dvi
R_UNZIPCMD /usr/bin/unzip
R_ZIPCMD /usr/bin/zip
RMARKDOWN_MATHJAX_PATH
/Applications/RStudio.app/Contents/Resources/resources/mathjax-26
RS_RPOSTBACK_PATH /Applications/RStudio.app/Contents/MacOS/rpostback
RS_SHARED_SECRET 6f83ef69-6b23-49d0-917a-e2aefba3b838
RSTUDIO 1
RSTUDIO_PANDOC /Applications/RStudio.app/Contents/MacOS/pandoc
RSTUDIO_SESSION_PORT 37083
RSTUDIO_USER_IDENTITY <<username>>
RSTUDIO_WINUTILS bin/winutils
SED /usr/bin/sed
SHELL /bin/bash
SSH_AUTH_SOCK /private/tmp/com.apple.launchd.K5yz0M2vBT/Listeners
TAR /usr/bin/tar
TMPDIR /var/folders/w6/ng3s613x6d9gyvfnl5l2yknjhtnwg4/T/
USER <<username>>
VIRTUAL_ENV /Users/<<username>>/.virtualenvs/r-tensorflow
XPC_FLAGS 0x0
XPC_SERVICE_NAME 0
I attempted a few different scenarios.
Launching and running a script from the command line with Rscript <
Problems - the plist suggestion seems to be hard to manage and it created conflicts with other environmental variables.
Possible work around, but not successful as of yet...
tpath <- Sys.getenv("PATH")
newpath <- paste0(tpath, ":/Developer/NVIDIA/CUDA-8.0/bin/lib")
Sys.setenv(PATH=newpath)
This does seem to append the path scope in R when evaluated with Sys.getenv()
. The dynamic libraries seems to be different. I don't see a "DYLD_LIBRARY_PATH" variable in R. There is a DYLD_FALLBACK_LIBRARY_PATH. Perhaps these function the same.
We made a little progress, but still something isn't quite right.
You actually can't set DYLD_LIBRARY_PATH during the R session, it needs to be set before the session starts (thus the .bash_profile or .plist requirement).
Another way to bootstrap without getting involve with the .plist is to launch RStudio from your bash terminal, e.g.
open -a RStudio
Well, look at that. That does the trick!
For those that may follow, could the installation instructions be updated some to indicate if you are using a Mac with a properly configured GPU configuration (i.e., Python will utilize Keras and Tensorflow), the necessary environment variables to the deep learning libraries may not load unless you launch RStudio from a command prompt using open -a RStudio
?
I've updated the docs on Mac GPU installation with this info, as well as a note on the fact that GPU support for Macs was discontinued in TF v1.1 and cuDNN 6.0: https://tensorflow.rstudio.com/tensorflow/articles/installation_gpu.html#mac-os-x
Setup - Macbook Pro with eGPU running properly on Sierra (10.12.6).
Anaconda installed with: Python 3.5.3 Tensorflow 1.1.0-gpu Keras 2.0.8 CUDA 8.0 and cuDNN 5.1 configured
My .bash_profile file has the following entries
I have two approaches for trying to run Keras currently. I'm looking to get the Rstudio interface running to give me a primary IDE.
Approaches without R interface: Approach 1 -- launch Spyder for editing from terminal with "spyder" command --Tensorflow runs. Keras runs. Models train quickly on GPU
Approach 2 -- launch Spyder from Anaconda dashboard -- Errors are generated saying Tensorflow is not located. This is in spite of seeing "tensorflow-gpu" version 1.1.0 listed as part of the packages under root
I've tried several iterations for getting keras to run in Rstudio. The following code seems to be a "by the book" attempt at the installation.
For take 2 at getting keras to install in r, I removed reticulate, keras, tensorflow and tfruns and gave it another go. This time, I tried to switch the python environment. The error changes to one that looks like the straight python approach 2 error listed above. For this reason, I'm thinking the issue might be related to environment variables and possible "confusion" of which python is which in Rstudio.
Second install log with error.
The note on "Library not loaded @rpath/libcublas.8.0.dylib" seems like the path or symbolic link. Is it as simple as not using .bash_profile and instead using .bashrc? Or, is it more complicated with python references. I've tried to research but I haven't found the solution.