search

t-kalinowski / deep-learning-with-R-2nd-edition-code

Code from the book "Deep Learning with R, 2nd Edition"
https://blogs.rstudio.com/ai/posts/2022-05-31-deep-learning-with-r-2e/
54 stars 22 forks source link

installation problem #3

Closed hseverac closed 1 year ago

hseverac commented 1 year ago

Hi I bought the Deep Learning with Python edition, though I have no knowledge of python, then I realized there was a Deep Learning with R edition, so I bought it also, as I have some proficiency with R (in RStudio)

Now I find myself blocked with Python setup problems. Could you help me ?

After following the instruction in install-r-tensorflow.R I can get nowhere.

at python <- reticulate::install_python("3.9:latest") I get failure to install python 3.9

python <- reticulate::install_python("3.9:latest")

  • '/usr/local/Cellar/pyenv/2.3.9/libexec/pyenv' install --skip-existing 3.9.16 /usr/local/Cellar/pyenv/2.3.9/libexec/pyenv-latest: line 39: printf: write error: Broken pipe python-build: use openssl@1.1 from homebrew python-build: use readline from homebrew Downloading Python-3.9.16.tar.xz... -> https://www.python.org/ftp/python/3.9.16/Python-3.9.16.tar.xz Installing Python-3.9.16... python-build: use readline from homebrew python-build: use zlib from xcode sdk

BUILD FAILED (OS X 12.6.3 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/m4/kxt6ryhx1qbgz4srx2mmybsw0000gn/T/python-build.20230216005716.54172 Results logged to /var/folders/m4/kxt6ryhx1qbgz4srx2mmybsw0000gn/T/python-build.20230216005716.54172.log

Last 10 log lines: checking for python3.9... python3.9 checking for --enable-universalsdk... no checking for --with-universal-archs... no checking MACHDEP... "darwin" checking for gcc... clang checking whether the C compiler works... no configure: error: in /var/folders/m4/kxt6ryhx1qbgz4srx2mmybsw0000gn/T/python-build.20230216005716.54172/Python-3.9.16': configure: error: C compiler cannot create executables Seeconfig.log' for more details make: *** No targets specified and no makefile found. Stop. Error: installation of Python 3.9.16 failed

If I impose a 3.9 version already installed by Home-brew a few monthes ago by python = "/usr/local/Cellar/python@3.9/3.9.16/Frameworks/Python.framework/Versions/3.9/bin/python3.9" I get error in installing tensorflow

  • '/Users/hseverac/.virtualenvs/r-reticulate/bin/python' -m pip install --upgrade --no-user --ignore-installed 'tensorflow==2.11.' 'tensorflow-hub' 'tensorflow-datasets' 'scipy' 'requests' 'Pillow' 'h5py' 'pandas' 'pydot' 'keras-tuner' 'ipython' 'kaggle' ERROR: Could not find a version that satisfies the requirement tensorflow==2.11. (from versions: 2.12.0rc0) ERROR: No matching distribution found for tensorflow==2.11. Error: Error installing package(s): "'tensorflow==2.11.'", "'tensorflow-hub'", "'tensorflow-datasets'", "'scipy'", "'requests'", "'Pillow'", "'h5py'", "'pandas'", "'pydot'", "'keras-tuner'", "'ipython'", "'kaggle'"

If I follow your other advice given elsewhere

install.packages("remotes") remotes::install_github(sprintf("rstudio/%s", c("reticulate", "tensorflow", "keras"))) reticulate::miniconda_uninstall() # start with a blank slate reticulate::install_miniconda() keras::install_keras()

It seems to be ok but then with library(tensorflow)

Error in value[3L] : Package ‘tensorflow’ version 2.11.0 cannot be unloaded: Error in unloadNamespace(package) : namespace ‘tensorflow’ is imported by ‘keras’ so cannot be unloaded

and with mnist <- dataset_mnist()

Error: Valid installation of TensorFlow not found.

Python environments searched for 'tensorflow' package: /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11

Python exception encountered: Traceback (most recent call last): File "/Library/Frameworks/R.framework/Versions/4.2/Resources/library/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook return _run_hook(name, _hook) ^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/R.framework/Versions/4.2/Resources/library/reticulate/python/rpytools/loader.py", line 93, in _run_hook module = hook() ^^^^^^ File "/Library/Frameworks/R.framework/Versions/4.2/Resources/library/reticulate/python/rpytools/loader.py", line 117, in _hook return _find_andload(name, import) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'tensorflow'

I must say I am lost with all these error messages and a little disappointed by all this setup problems. FYI, I have the 4 following versions of python installed :

/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11 /Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10 /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 /usr/local/Cellar/python@3.9/3.9.16/Frameworks/Python.framework/Versions/3.9/bin/python3.9

Thanks for your help.

t-kalinowski commented 1 year ago

Are you on an M1 mac? If yes, then presently tensorflow::install_tensorflow() is only compatible with miniconda python, which you can install with reticulate::install_miniconda(). Additionally, you must be running an Arm native build of R, not the x86 build running under Rosetta. Note the macOS tab section on https://tensorflow.rstudio.com/install/local_gpu:

You can test if you have an Arm build of R like this:

R.version$arch

If this returns "aarch64", you’re using the correct build of R. If instead it returns "x86_64", then you need to install a different build of R. You can install an Arm build of R by navigating to https://cloud.r-project.org/bin/macosx/ and selecting the appropriate package (e.g., “R-4.2.1-arm64.pkg”) or using an installation manager like rig and running rig add release.

This sequence of commands should work on an M1 mac:

reticulate::miniconda_uninstall() # start with a blank slate
reticulate::install_miniconda()
keras::install_keras()
hseverac commented 1 year ago

Thanks for answering. I'm on an Intel Mac. I have a big GPU still, a AMD Radeon Pro Vega 48. Does this question mean that you can't use tensorflow if you don't have a Nvidia card ?
I've seen conflicting info by people claiming they can tweak Cuda apps for Mac. Anyway, this is not my concern here as I 'm using it for learning purposes, and am not looking for speed, and could accept to fall for CPU use. I was wondering if I had rather a problem with my python version or a conflict with the tensorflow version

Here are the details of sessionInfo()

R version 4.2.2 (2022-10-31) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Monterey 12.6.3

Matrix products: default LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] keras_2.11.0.9000 tensorflow_2.11.0 reticulate_1.28-9000

loaded via a namespace (and not attached): [1] Rcpp_1.0.10 here_1.0.1 lattice_0.20-45 png_0.1-8 rprojroot_2.0.3 zeallot_0.1.0 withr_2.5.0 grid_4.2.2 R6_2.5.1
[10] jsonlite_1.8.4 magrittr_2.0.3 tfruns_1.5.1 cli_3.6.0 rlang_1.0.6 remotes_2.4.2 rstudioapi_0.14 whisker_0.4.1 Matrix_1.5-3
[19] generics_0.1.3 tools_4.2.2 compiler_4.2.2 base64enc_0.1-3

While 

library(tensorflow)
tf$config$list_physical_devices("GPU")

gets

Error: Valid installation of TensorFlow not found.

Python environments searched for 'tensorflow' package: /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11

Python exception encountered: Traceback (most recent call last): File "/Library/Frameworks/R.framework/Versions/4.2/Resources/library/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook return _run_hook(name, _hook) ^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/R.framework/Versions/4.2/Resources/library/reticulate/python/rpytools/loader.py", line 93, in _run_hook module = hook() ^^^^^^ File "/Library/Frameworks/R.framework/Versions/4.2/Resources/library/reticulate/python/rpytools/loader.py", line 117, in _hook return _find_andload(name, import) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'tensorflow'

I'm not familiar with the concept of python environments, so maybe the problem is here conda_list()

      name                                                           python

1 base /Users/hseverac/Library/r-miniconda/bin/python 2 r-reticulate /Users/hseverac/Library/r-miniconda/envs/r-reticulate/bin/python

and virtualenv_list()

[1] "r-reticulate"

hseverac commented 1 year ago

As a followup, as I believed the problem was with environments, I erased ~/.virtualenvs and /Users/hseverac/Library/r-miniconda then relaunched the installation 

install.packages("remotes")
remotes::install_github(sprintf("rstudio/%s", c("reticulate", "tensorflow", "keras")))
reticulate::miniconda_uninstall() # start with a blank slate
reticulate::install_miniconda()
keras::install_keras()

Which proceeded successfully.

However I stilll get the same error

mnist <- dataset_mnist() Error: Valid installation of TensorFlow not found.

If I try to force the environment I get this cryptic error message

use_condaenv("r-reticulate")

ERROR: The requested version of Python ('/Users/hseverac/Library/r-miniconda/envs/r-reticulate/bin/python') cannot be used, as another version of Python ('/usr/local/bin/python3') has already been initialized. Please restart the R session if you need to attach reticulate to a different version of Python. Error in use_python(python, required = required) : failed to initialize requested version of Python

I'm sorry to flood the thread with my Python problems, but I bought "Deep learning with R", and I m not supposed to know Python configuration astuces.

t-kalinowski commented 1 year ago

I'm on an Intel Mac. I have a big GPU still, a AMD Radeon Pro Vega 48.

This is going to be a very uphill battle to get a GPU to work on this machine.

Everything should work on your machine for CPU only tasks. Note, your version of Python installed is too recent, the latest Tensorflow supports only up to Python 3.10, not 3.11.

If you want TensorFlow to use a GPU, unfortunately your best option is to install Linux and get an Nvidia GPU.

Note, all these issues are orthogonal to whether you're using TensorFlow from R or Python, you would encounter the same issues if you're working in Python.

hseverac commented 1 year ago

Thanks for answering me on the GPU part. But as said, my main problem is that I bought a book advertised as Deep learning for R and I would like a minimal set of instruction guaranteeing I can use the example provided in the book on my machine (even on a CPU) without having to learn how to set up Python. If Tensorflow can't work with Python 3.11, there should be someone providing the necessary configuration setup allowing it to work. So far I spent maybe 5 hours using 2 sets of your recommandations : here, here

and it is not working. Python is working on my machine, the packages are loaded so there is something wrong in your configuration process specific to tensorfow, and I would appreciate you help me to solve it.

sessionInfo() R version 4.2.2 (2022-10-31) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Monterey 12.6.3 Matrix products: default LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base
other attached packages: [1] reticulate_1.28-9000 keras_2.11.0.9000 tensorflow_2.11.0.9000

t-kalinowski commented 1 year ago

ERROR: The requested version of Python ('/Users/hseverac/Library/r-miniconda/envs/r-reticulate/bin/python') cannot be used, as another version of Python ('/usr/local/bin/python3') has already been initialized. Please restart the R session if you need to attach reticulate

This indicates that reticulate was already bound to a python interperter when use_condaenv() was called.

hseverac commented 1 year ago

Thanks for helping.

I've been using a fresh R session all along. There is nothing in my .RProfile linked to Python.
The problem seems to be that I'm using Python 3.11 per default and I don't know how to change it.

library(reticulate)
use_python("/usr/local/bin/python3.9")

provides an error

ERROR: The requested version of Python ('/usr/local/bin/python3.9') cannot be used, as another version of Python ('/usr/local/bin/python3') has already been initialized. Please restart the R session if you need to attach reticulate to a different version of Python. Error in use_python("/usr/local/bin/python3.9") : failed to initialize requested version of Python

hseverac commented 1 year ago

We're progressing. I remembered you could choose the python interpretor in RStudio options. Which is now /usr/local/bin/python3.9.16

Now I get a new error

keras::install_keras() Error in get(Info[i, 1], envir = env) : lazy-load database '/Library/Frameworks/R.framework/Versions/4.2/Resources/library/tensorflow/R/tensorflow.rdb' is corrupt In addition: Warning message: In get(Info[i, 1], envir = env) : internal error -3 in R_decompress1

t-kalinowski commented 1 year ago

ERROR: The requested version of Python ('/usr/local/bin/python3.9') cannot be used, as another version of Python ('/usr/local/bin/python3') has already been initialized. Please restart the R session if you need to attach reticulate to a different version of Python.

This is indicating that something is still causing reticulate to initialize python before you interact with it. Are you activating the python repl in the RStudio IDE maybe? Some debugigng steps:

lazy-load database '/Library/Frameworks/R.framework/Versions/4.2/Resources/library/tensorflow/R/tensorflow.rdb' is corrupt

This happens if installation of the R package failed for some reason, most often due to being interrupted or due to file-locking contention. You can try to call remotes::install_github() again and that usually clears it up.

hseverac commented 1 year ago

I forced remote reinstallation through

install.packages("remotes") remotes::install_github(sprintf("rstudio/%s", c("reticulate", "tensorflow", "keras")),force = TRUE) reticulate::miniconda_uninstall() # start with a blank slate reticulate::install_miniconda() keras::install_keras()

which went OK ...except for the last line

keras::install_keras() Error: .onLoad failed in loadNamespace() for 'tensorflow', details: call: NULL error: ModuleNotFoundError: No module named 'tensorflow'

and sessionInfo() does not give any package attached , but [22] reticulate loaded via a namespace

R version 4.2.2 (2022-10-31) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Monterey 12.6.3

Matrix products: default LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages: [1] stats graphics grDevices utils datasets methods base

loaded via a namespace (and not attached): [1] Rcpp_1.0.10 compiler_4.2.2 prettyunits_1.1.1 base64enc_0.1-3 remotes_2.4.2 tools_4.2.2 zeallot_0.1.0
[8] pkgbuild_1.4.0 jsonlite_1.8.4 lattice_0.20-45 png_0.1-8 rlang_1.0.6 Matrix_1.5-3 cli_3.6.0
[15] rstudioapi_0.14 curl_5.0.0 withr_2.5.0 desc_1.4.2 generics_0.1.3 rprojroot_2.0.3 grid_4.2.2
[22] reticulate_1.28-9000 here_1.0.1 R6_2.5.1 processx_3.8.0 callr_3.7.3 magrittr_2.0.3 whisker_0.4.1
[29] ps_1.7.2 tfruns_1.5.1 crayon_1.5.2

hseverac commented 1 year ago

Seems I finally succeeded. The trick was to select in RStudio options: Python / Select / [automatically activate virtual environment ] ./virtualenvs/r-reticulate/bin/python3.9

Don't ask me why since I was expecting a miniconda envrt

At least

library(tensorflow) library(keras) mnist <- dataset_mnist()

is now downloading the data

Rosiethuypham commented 1 year ago

Hey, I have the same problem. Can you please tell me exactly what you did when you mentioning this:

"The trick was to select in RStudio options: Python / Select / [automatically activate virtual environment ] ./virtualenvs/r-reticulate/bin/python3.9"

t-kalinowski commented 1 year ago

Hi, the installation instruction (and underlying code) on https://rstudio.tensorflow.org/install were updated recently. To install, on all platforms, this should now be sufficient:

install.packages("remotes")
remotes::install_github("rstudio/keras")
reticulate::install_python()
keras::install_keras()