I've been trying to install Keras and Tensorflow from R following the instructions in the book, but I'm encountering an issue that I can't solve even following the discussion here.
First I installed the libraries on Windows using RStudio and everything worked perfectly. However, I only got it to work on CPU, not GPU. To facilitate the installation of dependencies to use the GPU I followed the instructions in the book and used Windows Subsystem for Linux to install Linux on my computer.
Once I did this, using Visual Studio Code with WSL to run the same code, the installation does not work. The end result is as follows:
Error: Valid installation of TensorFlow not found.
Python environments searched for 'tensorflow' package:
/home/dberm/.pyenv/versions/3.9.16/bin/python3.9
/usr/bin/python3.10
installing source package ‘keras’ ...
package ‘keras’ successfully unpacked and MD5 sums checked
using staged installation
R
inst
byte-compile and prepare package for lazy loading
help
* installing help indices
building package indices
installing vignettes
testing if installed package can be loaded from temporary location
testing if installed package can be loaded from final location
testing if installed package keeps a record of temporary installation path
DONE (keras)
The downloaded source packages are in
‘/tmp/RtmpDj8V9u/downloaded_packages’
'/home/dberm/.virtualenvs/r-reticulate/bin/python' -m pip install --upgrade --no-user --ignore-installed 'tensorflow==2.11.' 'tensorflow-hub' 'tensorflow-datasets' 'scipy' 'requests' 'Pillow' 'h5py' 'pandas' 'pydot'
Collecting tensorflow==2.11.
Using cached tensorflow-2.11.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (588.3 MB)
Collecting tensorflow-hub
Using cached tensorflow_hub-0.13.0-py2.py3-none-any.whl (100 kB)
Collecting tensorflow-datasets
Using cached tensorflow_datasets-4.8.3-py3-none-any.whl (5.4 MB)
Collecting scipy
Using cached scipy-1.10.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
Collecting requests
Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting Pillow
Using cached Pillow-9.4.0-cp39-cp39-manylinux_2_28_x86_64.whl (3.4 MB)
Collecting h5py
Using cached h5py-3.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
Collecting pandas
Using cached pandas-1.5.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)
Collecting pydot
Using cached pydot-1.4.2-py2.py3-none-any.whl (21 kB)
Collecting astunparse>=1.6.0
Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting opt-einsum>=2.3.2
Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB)
Collecting typing-extensions>=3.6.6
Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting google-pasta>=0.1.1
Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB)
Collecting grpcio<2.0,>=1.24.3
Using cached grpcio-1.51.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
Collecting tensorflow-estimator<2.12,>=2.11.0
Using cached tensorflow_estimator-2.11.0-py2.py3-none-any.whl (439 kB)
Collecting termcolor>=1.1.0
Using cached termcolor-2.2.0-py3-none-any.whl (6.6 kB)
Collecting libclang>=13.0.0
Using cached libclang-15.0.6.1-py2.py3-none-manylinux2010_x86_64.whl (21.5 MB)
Collecting packaging
Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting tensorboard<2.12,>=2.11
Using cached tensorboard-2.11.2-py3-none-any.whl (6.0 MB)
Collecting wrapt>=1.11.0
Using cached wrapt-1.15.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (78 kB)
Collecting flatbuffers>=2.0
Using cached flatbuffers-23.3.3-py2.py3-none-any.whl (26 kB)
Collecting tensorflow-io-gcs-filesystem>=0.23.1
Using cached tensorflow_io_gcs_filesystem-0.31.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.4 MB)
Collecting numpy>=1.20
Using cached numpy-1.24.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
Collecting protobuf<3.20,>=3.9.2
Using cached protobuf-3.19.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Collecting absl-py>=1.0.0
Using cached absl_py-1.4.0-py3-none-any.whl (126 kB)
Collecting keras<2.12,>=2.11.0
Using cached keras-2.11.0-py2.py3-none-any.whl (1.7 MB)
Collecting setuptools
Using cached setuptools-67.6.0-py3-none-any.whl (1.1 MB)
Collecting gast<=0.4.0,>=0.2.1
Using cached gast-0.4.0-py3-none-any.whl (9.8 kB)
Collecting six>=1.12.0
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting promise
Using cached promise-2.3.tar.gz (19 kB)
Preparing metadata (setup.py) ... done
Collecting tqdm
Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting dm-tree
Using cached dm_tree-0.1.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (153 kB)
Collecting tensorflow-metadata
Using cached tensorflow_metadata-1.12.0-py3-none-any.whl (52 kB)
Collecting toml
Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting etils[enp,epath]>=0.9.0
Using cached etils-1.1.1-py3-none-any.whl (115 kB)
Collecting psutil
Using cached psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)
Collecting click
Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting charset-normalizer<4,>=2
Using cached charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
Collecting idna<4,>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Collecting pytz>=2020.1
Using cached pytz-2022.7.1-py2.py3-none-any.whl (499 kB)
Collecting python-dateutil>=2.8.1
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pyparsing>=2.1.4
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting wheel<1.0,>=0.23.0
Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
Collecting zipp
Using cached zipp-3.15.0-py3-none-any.whl (6.8 kB)
Collecting importlib_resources
Using cached importlib_resources-5.12.0-py3-none-any.whl (36 kB)
Collecting tensorboard-data-server<0.7.0,>=0.6.0
Using cached tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
Collecting markdown>=2.6.8
Using cached Markdown-3.4.2-py3-none-any.whl (93 kB)
Collecting tensorboard-plugin-wit>=1.6.0
Using cached tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
Collecting google-auth<3,>=1.6.3
Using cached google_auth-2.16.2-py2.py3-none-any.whl (177 kB)
Collecting google-auth-oauthlib<0.5,>=0.4.1
Using cached google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting werkzeug>=1.0.1
Using cached Werkzeug-2.2.3-py3-none-any.whl (233 kB)
Collecting googleapis-common-protos<2,>=1.52.0
Using cached googleapis_common_protos-1.59.0-py2.py3-none-any.whl (223 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.9-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting cachetools<6.0,>=2.0.0
Using cached cachetools-5.3.0-py3-none-any.whl (9.3 kB)
Collecting requests-oauthlib>=0.7.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting importlib-metadata>=4.4
Using cached importlib_metadata-6.1.0-py3-none-any.whl (21 kB)
Collecting MarkupSafe>=2.1.1
Using cached MarkupSafe-2.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting pyasn1<0.5.0,>=0.4.6
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.2-py3-none-any.whl (151 kB)
Building wheels for collected packages: promise
Building wheel for promise (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [34 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-install-c9bisgzm/promise_af6168e0cb2e42adaeca9a8025bee6aa/setup.py", line 28, in
setup(
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/init.py", line 108, in setup
return distutils.core.setup(**attrs)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 172, in setup
ok = dist.parse_command_line()
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 475, in parse_command_line
args = self._parse_command_opts(parser, args)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/dist.py", line 1119, in _parse_command_opts
nargs = _Distribution._parse_command_opts(self, parser, args)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 534, in _parse_command_opts
cmd_class = self.get_command_class(command)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/dist.py", line 966, in get_command_class
self.cmdclass[command] = cmdclass = ep.load()
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/init.py", line 208, in load
module = import_module(match.group('module'))
File "/home/dberm/.pyenv/versions/3.9.16/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 28, in
from .macosx_libfile import calculate_macosx_platform_tag
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/wheel/macosx_libfile.py", line 43, in
import ctypes
File "/home/dberm/.pyenv/versions/3.9.16/lib/python3.9/ctypes/init.py", line 8, in
from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for promise
Running setup.py clean for promise
Failed to build promise
Installing collected packages: tensorboard-plugin-wit, pytz, pyasn1, libclang, flatbuffers, dm-tree, zipp, wrapt, wheel, urllib3, typing-extensions, tqdm, toml, termcolor, tensorflow-io-gcs-filesystem, tensorflow-estimator, tensorboard-data-server, six, setuptools, rsa, pyparsing, pyasn1-modules, psutil, protobuf, Pillow, packaging, oauthlib, numpy, MarkupSafe, keras, idna, grpcio, gast, etils, click, charset-normalizer, certifi, cachetools, absl-py, werkzeug, tensorflow-hub, scipy, requests, python-dateutil, pydot, promise, opt-einsum, importlib_resources, importlib-metadata, h5py, googleapis-common-protos, google-pasta, google-auth, astunparse, tensorflow-metadata, requests-oauthlib, pandas, markdown, tensorflow-datasets, google-auth-oauthlib, tensorboard, tensorflow
Running setup.py install for promise ... done
DEPRECATION: promise was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. pip 23.1 will enforce this behaviour change. A possible replacement is to fix the wheel build issue reported above. Discussion can be found at https://github.com/pypa/pip/issues/8368
Successfully installed MarkupSafe-2.1.2 Pillow-9.4.0 absl-py-1.4.0 astunparse-1.6.3 cachetools-5.3.0 certifi-2022.12.7 charset-normalizer-3.1.0 click-8.1.3 dm-tree-0.1.8 etils-1.1.1 flatbuffers-23.3.3 gast-0.4.0 google-auth-2.16.2 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 googleapis-common-protos-1.59.0 grpcio-1.51.3 h5py-3.8.0 idna-3.4 importlib-metadata-6.1.0 importlib_resources-5.12.0 keras-2.11.0 libclang-15.0.6.1 markdown-3.4.2 numpy-1.24.2 oauthlib-3.2.2 opt-einsum-3.3.0 packaging-23.0 pandas-1.5.3 promise-2.3 protobuf-3.19.6 psutil-5.9.4 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydot-1.4.2 pyparsing-3.0.9 python-dateutil-2.8.2 pytz-2022.7.1 requests-2.28.2 requests-oauthlib-1.3.1 rsa-4.9 scipy-1.10.1 setuptools-67.6.0 six-1.16.0 tensorboard-2.11.2 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-2.11.1 tensorflow-datasets-4.8.3 tensorflow-estimator-2.11.0 tensorflow-hub-0.13.0 tensorflow-io-gcs-filesystem-0.31.0 tensorflow-metadata-1.12.0 termcolor-2.2.0 toml-0.10.2 tqdm-4.65.0 typing-extensions-4.5.0 urllib3-1.26.15 werkzeug-2.2.3 wheel-0.40.0 wrapt-1.15.0 zipp-3.15.0
Installation complete.
tensorflow::tf_config()
Valid installation of TensorFlow not found.
Python environments searched for 'tensorflow' package:
/home/dberm/.pyenv/versions/3.9.16/bin/python3.9
/usr/bin/python3.10
Python exception encountered:
Traceback (most recent call last):
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return _find_andload(name, import)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/tensorflow/init.py", line 37, in
from tensorflow.python.tools import module_util as _module_util
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return _find_andload(name, import)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return _find_andload(name, import)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/tensorflow/python/init.py", line 24, in
import ctypes
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return _find_andload(name, import)
File "/home/dberm/.pyenv/versions/3.9.16/lib/python3.9/ctypes/init.py", line 8, in
from _ctypes import Union, Structure, Array
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return _find_andload(name, import)
ModuleNotFoundError: No module named '_ctypes'
You can install TensorFlow using the install_tensorflow() function.
If I try to use tensorflow::install_tensorflow() before tensorflow::tf_config() the result is the same. I also tried the instructions from the RStudio website, with the same results:
Additionally, you must be running an Arm native build of R, not the x86 build running under Rosetta.
Which I don't know if applies to my case or is just for Intel Mac.
I don't know what else to try as I have been several days stuck here and I lack the required Python knowledge to make any progress. Any help would be appreciated.
I've been trying to install Keras and Tensorflow from R following the instructions in the book, but I'm encountering an issue that I can't solve even following the discussion here.
First I installed the libraries on Windows using RStudio and everything worked perfectly. However, I only got it to work on CPU, not GPU. To facilitate the installation of dependencies to use the GPU I followed the instructions in the book and used Windows Subsystem for Linux to install Linux on my computer.
Once I did this, using Visual Studio Code with WSL to run the same code, the installation does not work. The end result is as follows:
My full code so far:
install.packages("keras")library(reticulate)virtualenv_create("r-reticulate", python = install_python())tensorflow::tf_config()If I try to use
tensorflow::install_tensorflow()beforetensorflow::tf_config()the result is the same. I also tried the instructions from the RStudio website, with the same results:As far as I can tell by the output,
tensorflow::tf_config()is looking for Tensorflow here:while Tensorflor should be in
envname = "r-reticulate", but I'm not used to work with Python enviroments so I don't know to change this behaviour.I also read here this comment:
Which I don't know if applies to my case or is just for Intel Mac.
I don't know what else to try as I have been several days stuck here and I lack the required Python knowledge to make any progress. Any help would be appreciated.