search

rocker-org / rocker-versioned2

Run current & prior versions of R using docker. rocker/r-ver, rocker/rstudio, rocker/shiny, rocker/tidyverse, and so on.
https://rocker-project.org
GNU General Public License v2.0
421 stars 181 forks source link

rocker/ml:latest fails to install tensorflow due to not finding its own venv #880

Open sneumann opened 4 days ago

sneumann commented 4 days ago

Container image name

rocker/ml:latest

Container image digest

rocker/ml@sha256:327295f35a097a7ee2fec46804e0a285057c9918df1e4e12470e915e29e7de40

What operating system are you seeing the problem on?

Linux

System information

Docker version 24.0.7-ce, build 311b9ff0aa93 on openSUSE VERSION = 15.5

Bug description

Hi, I am installing keras/tensorflow in rocker/ml. Installation runs fine, From python I can activate the venv /root/.virtualenvs/r-tensorflow and python can import tensorflow, and even shows some GPUs. Issue happens when using library(tensorflow), because ModuleNotFoundError: No module named 'tensorflow'.

On an older rocker/rstudio based container with R-4.4.1 I get

R version 4.4.1 (2024-06-14) -- "Race for Your Life"
...
> library(tensorflow)
> reticulate::py_config()
python:         /root/.virtualenvs/r-tensorflow/bin/python
...
tensorflow:     /root/.virtualenvs/r-tensorflow/lib/python3.10/site-packages/tensorflo
NOTE: Python version was forced by import("tensorflow")

while the non-working rocker/ml also installed into /root/.virtualenvs/r-tensorflow

Virtual environment 'r-tensorflow' successfully created.
Using virtual environment 'r-tensorflow' ...
+ /root/.virtualenvs/r-tensorflow/bin/python -m pip install --upgrade --no-user 'numpy<2' 'tensorflow[and-cuda]==2.15.*' tensorflow-hub tensorflow-datasets scipy requests Pillow h5py pandas pydot
...
Successfully installed MarkupSafe-3.0.2 Pillow-11.0.0 absl-py-2.1.0 array-record-0.5.1 astunparse-1.6.3 cachetools-5.5.0 certifi-2024.8.30 charset-normalizer-3.4.0 click-8.1.7 dm-tree-0.1.8 docstring-parser-0.16 etils-1.11.0 flatbuffers-24.3.25 fsspec-2024.10.0 gast-0.6.0 google-auth-2.36.0 google-auth-oauthlib-1.2.1 google-pasta-0.2.0 grpcio-1.68.1 h5py-3.12.1 idna-3.10 immutabledict-4.2.1 importlib_resources-6.4.5 keras-2.15.0 libclang-18.1.1 markdown-3.7 ml-dtypes-0.3.2 numpy-1.26.4 nvidia-cublas-cu12-12.2.5.6 nvidia-cuda-cupti-cu12-12.2.142 nvidia-cuda-nvcc-cu12-12.2.140 nvidia-cuda-nvrtc-cu12-12.2.140 nvidia-cuda-runtime-cu12-12.2.140 nvidia-cudnn-cu12-8.9.4.25 nvidia-cufft-cu12-11.0.8.103 nvidia-curand-cu12-10.3.3.141 nvidia-cusolver-cu12-11.5.2.141 nvidia-cusparse-cu12-12.1.2.141 nvidia-nccl-cu12-2.16.5 nvidia-nvjitlink-cu12-12.2.140 oauthlib-3.2.2 opt-einsum-3.4.0 packaging-24.2 pandas-2.2.3 promise-2.3 protobuf-3.20.3 psutil-6.1.0 pyarrow-18.1.0 pyasn1-0.6.1 pyasn1-modules-0.4.1 pydot-3.0.3 pyparsing-3.2.0 python-dateutil-2.9.0.post0 pytz-2024.2 requests-2.32.3 requests-oauthlib-2.0.0 rsa-4.9 scipy-1.14.1 simple-parsing-0.1.6 six-1.17.0 tensorboard-2.15.2 tensorboard-data-server-0.7.2 tensorflow-2.15.1 tensorflow-datasets-4.9.7 tensorflow-estimator-2.15.0 tensorflow-hub-0.16.1 tensorflow-io-gcs-filesystem-0.37.1 tensorflow-metadata-1.16.1 termcolor-2.5.0 tf-keras-2.15.1 toml-0.10.2 tqdm-4.67.1 typing-extensions-4.12.2 tzdata-2024.2 urllib3-2.2.3 werkzeug-3.1.3 wrapt-1.14.1 zipp-3.21.0
creating symlinks:
- '/root/.virtualenvs/r-tensorflow/lib/python3.10/site-packages/tensorflow/libcublasLt.so.12' -> '../nvidia/cublas/lib/libcublasLt.so.12'
- '/root/.virtualenvs/r-tensorflow/lib/python3.10/site-packages/tensorflow/libcublas.so.12' -> '../nvidia/cublas/lib/libcublas.so.12'
...
Installation complete.

but a different python venv /opt/venv/ is used when I then want to load the library:

R version 4.4.1 (2024-06-14) -- "Race for Your Life"
...
> library(tensorflow)
> reticulate::py_config()
python:         /opt/venv/bin/python
libpython:      /usr/lib/python3.10/config-3.10-x86_64-linux-gnu/libpython3.10.so
pythonhome:     /opt/venv:/opt/venv
version:        3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
numpy:           [NOT FOUND]
tensorflow:     [NOT FOUND]

NOTE: Python version was forced by VIRTUAL_ENV

=> How to inform reticulate/tensorflow to use the correct venv ? I upgraded reticulate from 1.37 -> 1.40 which is the one in the working installation, same error. Both images are based on 22.04, both use python 3.10.12.

Yours, Steffen

How to reproduce this bug?

docker run --rm -it rocker/ml R -e 'remotes::install_github("rstudio/tensorflow", upgrade="never"); library(tensorflow); install.packages("keras"); library(keras); install_keras(); model.sla <- keras_model_sequential(input_shape = c(NULL, 12906, 1))'

Full build log:

docker run --rm -it rocker/ml R -e 'remotes::install_github("rstudio/tensorflow", upgrade="never"); library(tensorflow); install.packages("keras"); library(keras); install_keras(); model.sla <- keras_model_sequential(input_shape = c(NULL, 12906, 1))'

==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .

R version 4.4.1 (2024-06-14) -- "Race for Your Life"
Copyright (C) 2024 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> remotes::install_github("rstudio/tensorflow", upgrade="never"); library(tensorflow); install.packages("keras"); library(keras); install_keras(); model.sla <- keras_model_sequential(input_shape = c(NULL, 12906, 1))
Downloading GitHub repo rstudio/tensorflow@HEAD
Installing 3 packages: config, tfautograph, tfruns
Installing packages into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'https://p3m.dev/cran/__linux__/jammy/latest/src/contrib/config_0.3.2.tar.gz'
Content type 'binary/octet-stream' length 102935 bytes (100 KB)
==================================================
downloaded 100 KB

trying URL 'https://p3m.dev/cran/__linux__/jammy/latest/src/contrib/tfautograph_0.3.2.tar.gz'
Content type 'binary/octet-stream' length 162247 bytes (158 KB)
==================================================
downloaded 158 KB

trying URL 'https://p3m.dev/cran/__linux__/jammy/latest/src/contrib/tfruns_1.5.3.tar.gz'
Content type 'binary/octet-stream' length 1486076 bytes (1.4 MB)
==================================================
downloaded 1.4 MB

* installing *binary* package ‘config’ ...
* DONE (config)
* installing *binary* package ‘tfautograph’ ...
* DONE (tfautograph)
* installing *binary* package ‘tfruns’ ...
* DONE (tfruns)

The downloaded source packages are in
        ‘/tmp/RtmpFTTYLw/downloaded_packages’
── R CMD build ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔  checking for file ‘/tmp/RtmpFTTYLw/remotes12c84dcc0/rstudio-tensorflow-e26a203/DESCRIPTION’ ...
─  preparing ‘tensorflow’:
✔  checking DESCRIPTION meta-information ...
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
─  building ‘tensorflow_2.16.0.9000.tar.gz’

Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
* installing *source* package ‘tensorflow’ ...
** using staged installation
** R
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (tensorflow)
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
also installing the dependency ‘zeallot’

trying URL 'https://p3m.dev/cran/__linux__/jammy/latest/src/contrib/zeallot_0.1.0.tar.gz'
Content type 'binary/octet-stream' length 59168 bytes (57 KB)
==================================================
downloaded 57 KB

trying URL 'https://p3m.dev/cran/__linux__/jammy/latest/src/contrib/keras_2.15.0.tar.gz'
Content type 'binary/octet-stream' length 3263440 bytes (3.1 MB)
==================================================
downloaded 3.1 MB

* installing *binary* package ‘zeallot’ ...
* DONE (zeallot)
* installing *binary* package ‘keras’ ...
* DONE (keras)

The downloaded source packages are in
        ‘/tmp/RtmpFTTYLw/downloaded_packages’
sh: 1: lspci: not found
sh: 1: nvidia-smi: not found
Using Python: /usr/bin/python3.10
Creating virtual environment 'r-tensorflow' ... 
+ /usr/bin/python3.10 -m venv /root/.virtualenvs/r-tensorflow
Done!
Installing packages: pip, wheel, setuptools
+ /root/.virtualenvs/r-tensorflow/bin/python -m pip install --upgrade pip wheel setuptools
Requirement already satisfied: pip in /root/.virtualenvs/r-tensorflow/lib/python3.10/site-packages (22.0.2)
Collecting pip
  Downloading pip-24.3.1-py3-none-any.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 13.6 MB/s eta 0:00:00
Collecting wheel
  Downloading wheel-0.45.1-py3-none-any.whl (72 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 72.5/72.5 KB 27.2 MB/s eta 0:00:00
Requirement already satisfied: setuptools in /root/.virtualenvs/r-tensorflow/lib/python3.10/site-packages (59.6.0)
Collecting setuptools
  Downloading setuptools-75.6.0-py3-none-any.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 5.4 MB/s eta 0:00:00
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 59.6.0
    Uninstalling setuptools-59.6.0:
      Successfully uninstalled setuptools-59.6.0
  Attempting uninstall: pip
    Found existing installation: pip 22.0.2
    Uninstalling pip-22.0.2:
      Successfully uninstalled pip-22.0.2
Successfully installed pip-24.3.1 setuptools-75.6.0 wheel-0.45.1
Virtual environment 'r-tensorflow' successfully created.
Using virtual environment 'r-tensorflow' ...
+ /root/.virtualenvs/r-tensorflow/bin/python -m pip install --upgrade --no-user 'numpy<2' 'tensorflow==2.15.*' tensorflow-hub tensorflow-datasets scipy requests Pillow h5py pandas pydot
Collecting numpy<2
  Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
Collecting tensorflow==2.15.*
  Downloading tensorflow-2.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
Collecting tensorflow-hub
  Downloading tensorflow_hub-0.16.1-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting tensorflow-datasets
  Downloading tensorflow_datasets-4.9.7-py3-none-any.whl.metadata (9.6 kB)
Collecting scipy
  Downloading scipy-1.14.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
Collecting requests
  Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting Pillow
  Downloading pillow-11.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (9.1 kB)
Collecting h5py
  Downloading h5py-3.12.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.5 kB)
Collecting pandas
  Downloading pandas-2.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (89 kB)
Collecting pydot
  Downloading pydot-3.0.3-py3-none-any.whl.metadata (10 kB)
Collecting absl-py>=1.0.0 (from tensorflow==2.15.*)
  Downloading absl_py-2.1.0-py3-none-any.whl.metadata (2.3 kB)
Collecting astunparse>=1.6.0 (from tensorflow==2.15.*)
  Downloading astunparse-1.6.3-py2.py3-none-any.whl.metadata (4.4 kB)
Collecting flatbuffers>=23.5.26 (from tensorflow==2.15.*)
  Downloading flatbuffers-24.3.25-py2.py3-none-any.whl.metadata (850 bytes)
Collecting gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 (from tensorflow==2.15.*)
  Downloading gast-0.6.0-py3-none-any.whl.metadata (1.3 kB)
Collecting google-pasta>=0.1.1 (from tensorflow==2.15.*)
  Downloading google_pasta-0.2.0-py3-none-any.whl.metadata (814 bytes)
Collecting libclang>=13.0.0 (from tensorflow==2.15.*)
  Downloading libclang-18.1.1-py2.py3-none-manylinux2010_x86_64.whl.metadata (5.2 kB)
Collecting ml-dtypes~=0.3.1 (from tensorflow==2.15.*)
  Downloading ml_dtypes-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (20 kB)
Collecting opt-einsum>=2.3.2 (from tensorflow==2.15.*)
  Downloading opt_einsum-3.4.0-py3-none-any.whl.metadata (6.3 kB)
Collecting packaging (from tensorflow==2.15.*)
  Downloading packaging-24.2-py3-none-any.whl.metadata (3.2 kB)
Collecting protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 (from tensorflow==2.15.*)
  Downloading protobuf-4.25.5-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes)
Requirement already satisfied: setuptools in /root/.virtualenvs/r-tensorflow/lib/python3.10/site-packages (from tensorflow==2.15.*) (75.6.0)
Collecting six>=1.12.0 (from tensorflow==2.15.*)
  Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Collecting termcolor>=1.1.0 (from tensorflow==2.15.*)
  Downloading termcolor-2.5.0-py3-none-any.whl.metadata (6.1 kB)
Collecting typing-extensions>=3.6.6 (from tensorflow==2.15.*)
  Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Collecting wrapt<1.15,>=1.11.0 (from tensorflow==2.15.*)
  Downloading wrapt-1.14.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
Collecting tensorflow-io-gcs-filesystem>=0.23.1 (from tensorflow==2.15.*)
  Downloading tensorflow_io_gcs_filesystem-0.37.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (14 kB)
Collecting grpcio<2.0,>=1.24.3 (from tensorflow==2.15.*)
  Downloading grpcio-1.68.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.9 kB)
Collecting tensorboard<2.16,>=2.15 (from tensorflow==2.15.*)
  Downloading tensorboard-2.15.2-py3-none-any.whl.metadata (1.7 kB)
Collecting tensorflow-estimator<2.16,>=2.15.0 (from tensorflow==2.15.*)
  Downloading tensorflow_estimator-2.15.0-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting keras<2.16,>=2.15.0 (from tensorflow==2.15.*)
  Downloading keras-2.15.0-py3-none-any.whl.metadata (2.4 kB)
Collecting tf-keras>=2.14.1 (from tensorflow-hub)
  Downloading tf_keras-2.18.0-py3-none-any.whl.metadata (1.6 kB)
Collecting click (from tensorflow-datasets)
  Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting dm-tree (from tensorflow-datasets)
  Downloading dm_tree-0.1.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.9 kB)
Collecting immutabledict (from tensorflow-datasets)
  Downloading immutabledict-4.2.1-py3-none-any.whl.metadata (3.5 kB)
Collecting promise (from tensorflow-datasets)
  Downloading promise-2.3.tar.gz (19 kB)
  Preparing metadata (setup.py) ... done
Collecting psutil (from tensorflow-datasets)
  Downloading psutil-6.1.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (22 kB)
Collecting pyarrow (from tensorflow-datasets)
  Downloading pyarrow-18.1.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.3 kB)
Collecting simple-parsing (from tensorflow-datasets)
  Downloading simple_parsing-0.1.6-py3-none-any.whl.metadata (7.3 kB)
Collecting tensorflow-metadata (from tensorflow-datasets)
  Downloading tensorflow_metadata-1.16.1-py3-none-any.whl.metadata (2.4 kB)
Collecting toml (from tensorflow-datasets)
  Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB)
Collecting tqdm (from tensorflow-datasets)
  Downloading tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)
Collecting array-record>=0.5.0 (from tensorflow-datasets)
  Downloading array_record-0.5.1-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (699 bytes)
Collecting etils>=1.6.0 (from etils[edc,enp,epath,epy,etree]>=1.6.0; python_version < "3.11"->tensorflow-datasets)
  Downloading etils-1.11.0-py3-none-any.whl.metadata (6.5 kB)
Collecting charset-normalizer<4,>=2 (from requests)
  Downloading charset_normalizer-3.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (34 kB)
Collecting idna<4,>=2.5 (from requests)
  Downloading idna-3.10-py3-none-any.whl.metadata (10 kB)
Collecting urllib3<3,>=1.21.1 (from requests)
  Downloading urllib3-2.2.3-py3-none-any.whl.metadata (6.5 kB)
Collecting certifi>=2017.4.17 (from requests)
  Downloading certifi-2024.8.30-py3-none-any.whl.metadata (2.2 kB)
Collecting python-dateutil>=2.8.2 (from pandas)
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytz>=2020.1 (from pandas)
  Downloading pytz-2024.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting tzdata>=2022.7 (from pandas)
  Downloading tzdata-2024.2-py2.py3-none-any.whl.metadata (1.4 kB)
Collecting pyparsing>=3.0.9 (from pydot)
  Downloading pyparsing-3.2.0-py3-none-any.whl.metadata (5.0 kB)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /root/.virtualenvs/r-tensorflow/lib/python3.10/site-packages (from astunparse>=1.6.0->tensorflow==2.15.*) (0.45.1)
Collecting fsspec (from etils[edc,enp,epath,epy,etree]>=1.6.0; python_version < "3.11"->tensorflow-datasets)
  Downloading fsspec-2024.10.0-py3-none-any.whl.metadata (11 kB)
Collecting importlib_resources (from etils[edc,enp,epath,epy,etree]>=1.6.0; python_version < "3.11"->tensorflow-datasets)
  Downloading importlib_resources-6.4.5-py3-none-any.whl.metadata (4.0 kB)
Collecting zipp (from etils[edc,enp,epath,epy,etree]>=1.6.0; python_version < "3.11"->tensorflow-datasets)
  Downloading zipp-3.21.0-py3-none-any.whl.metadata (3.7 kB)
Collecting google-auth<3,>=1.6.3 (from tensorboard<2.16,>=2.15->tensorflow==2.15.*)
  Downloading google_auth-2.36.0-py2.py3-none-any.whl.metadata (4.7 kB)
Collecting google-auth-oauthlib<2,>=0.5 (from tensorboard<2.16,>=2.15->tensorflow==2.15.*)
  Downloading google_auth_oauthlib-1.2.1-py2.py3-none-any.whl.metadata (2.7 kB)
Collecting markdown>=2.6.8 (from tensorboard<2.16,>=2.15->tensorflow==2.15.*)
  Downloading Markdown-3.7-py3-none-any.whl.metadata (7.0 kB)
Collecting tensorboard-data-server<0.8.0,>=0.7.0 (from tensorboard<2.16,>=2.15->tensorflow==2.15.*)
  Downloading tensorboard_data_server-0.7.2-py3-none-manylinux_2_31_x86_64.whl.metadata (1.1 kB)
Collecting werkzeug>=1.0.1 (from tensorboard<2.16,>=2.15->tensorflow==2.15.*)
  Downloading werkzeug-3.1.3-py3-none-any.whl.metadata (3.7 kB)
INFO: pip is looking at multiple versions of tf-keras to determine which version is compatible with other requirements. This could take a while.
Collecting tf-keras>=2.14.1 (from tensorflow-hub)
  Downloading tf_keras-2.17.0-py3-none-any.whl.metadata (1.6 kB)
  Downloading tf_keras-2.16.0-py3-none-any.whl.metadata (1.6 kB)
  Downloading tf_keras-2.15.1-py3-none-any.whl.metadata (1.7 kB)
Collecting docstring-parser<1.0,>=0.15 (from simple-parsing->tensorflow-datasets)
  Downloading docstring_parser-0.16-py3-none-any.whl.metadata (3.0 kB)
Collecting protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 (from tensorflow==2.15.*)
  Downloading protobuf-3.20.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata (679 bytes)
Collecting cachetools<6.0,>=2.0.0 (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow==2.15.*)
  Downloading cachetools-5.5.0-py3-none-any.whl.metadata (5.3 kB)
Collecting pyasn1-modules>=0.2.1 (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow==2.15.*)
  Downloading pyasn1_modules-0.4.1-py3-none-any.whl.metadata (3.5 kB)
Collecting rsa<5,>=3.1.4 (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow==2.15.*)
  Downloading rsa-4.9-py3-none-any.whl.metadata (4.2 kB)
Collecting requests-oauthlib>=0.7.0 (from google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow==2.15.*)
  Downloading requests_oauthlib-2.0.0-py2.py3-none-any.whl.metadata (11 kB)
Collecting MarkupSafe>=2.1.1 (from werkzeug>=1.0.1->tensorboard<2.16,>=2.15->tensorflow==2.15.*)
  Downloading MarkupSafe-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Collecting pyasn1<0.7.0,>=0.4.6 (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow==2.15.*)
  Downloading pyasn1-0.6.1-py3-none-any.whl.metadata (8.4 kB)
Collecting oauthlib>=3.0.0 (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow==2.15.*)
  Downloading oauthlib-3.2.2-py3-none-any.whl.metadata (7.5 kB)
Downloading tensorflow-2.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (475.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 475.2/475.2 MB 25.8 MB/s eta 0:00:00
Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 26.1 MB/s eta 0:00:00
Downloading tensorflow_hub-0.16.1-py2.py3-none-any.whl (30 kB)
Downloading tensorflow_datasets-4.9.7-py3-none-any.whl (5.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.3/5.3 MB 27.1 MB/s eta 0:00:00
Downloading scipy-1.14.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (41.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.2/41.2 MB 27.1 MB/s eta 0:00:00
Downloading requests-2.32.3-py3-none-any.whl (64 kB)
Downloading pillow-11.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.4 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.4/4.4 MB 28.6 MB/s eta 0:00:00
Downloading h5py-3.12.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.3/5.3 MB 28.8 MB/s eta 0:00:00
Downloading pandas-2.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.1/13.1 MB 24.7 MB/s eta 0:00:00
Downloading pydot-3.0.3-py3-none-any.whl (35 kB)
Downloading absl_py-2.1.0-py3-none-any.whl (133 kB)
Downloading array_record-0.5.1-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (2.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 20.2 MB/s eta 0:00:00
Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Downloading certifi-2024.8.30-py3-none-any.whl (167 kB)
Downloading charset_normalizer-3.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)
Downloading etils-1.11.0-py3-none-any.whl (165 kB)
Downloading flatbuffers-24.3.25-py2.py3-none-any.whl (26 kB)
Downloading gast-0.6.0-py3-none-any.whl (21 kB)
Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
Downloading grpcio-1.68.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.9/5.9 MB 21.4 MB/s eta 0:00:00
Downloading idna-3.10-py3-none-any.whl (70 kB)
Downloading keras-2.15.0-py3-none-any.whl (1.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 20.5 MB/s eta 0:00:00
Downloading libclang-18.1.1-py2.py3-none-manylinux2010_x86_64.whl (24.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.5/24.5 MB 23.1 MB/s eta 0:00:00
Downloading ml_dtypes-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 22.3 MB/s eta 0:00:00
Downloading opt_einsum-3.4.0-py3-none-any.whl (71 kB)
Downloading pyparsing-3.2.0-py3-none-any.whl (106 kB)
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Downloading pytz-2024.2-py2.py3-none-any.whl (508 kB)
Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
Downloading tensorboard-2.15.2-py3-none-any.whl (5.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 22.1 MB/s eta 0:00:00
Downloading tensorflow_estimator-2.15.0-py2.py3-none-any.whl (441 kB)
Downloading tensorflow_io_gcs_filesystem-0.37.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.1/5.1 MB 23.1 MB/s eta 0:00:00
Downloading termcolor-2.5.0-py3-none-any.whl (7.8 kB)
Downloading tf_keras-2.15.1-py3-none-any.whl (1.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 22.5 MB/s eta 0:00:00
Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Downloading tzdata-2024.2-py2.py3-none-any.whl (346 kB)
Downloading urllib3-2.2.3-py3-none-any.whl (126 kB)
Downloading wrapt-1.14.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (77 kB)
Downloading click-8.1.7-py3-none-any.whl (97 kB)
Downloading dm_tree-0.1.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (152 kB)
Downloading immutabledict-4.2.1-py3-none-any.whl (4.7 kB)
Downloading packaging-24.2-py3-none-any.whl (65 kB)
Downloading psutil-6.1.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (287 kB)
Downloading pyarrow-18.1.0-cp310-cp310-manylinux_2_28_x86_64.whl (40.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.1/40.1 MB 25.1 MB/s eta 0:00:00
Downloading simple_parsing-0.1.6-py3-none-any.whl (112 kB)
Downloading tensorflow_metadata-1.16.1-py3-none-any.whl (28 kB)
Downloading protobuf-3.20.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 20.5 MB/s eta 0:00:00
Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Downloading tqdm-4.67.1-py3-none-any.whl (78 kB)
Downloading docstring_parser-0.16-py3-none-any.whl (36 kB)
Downloading google_auth-2.36.0-py2.py3-none-any.whl (209 kB)
Downloading google_auth_oauthlib-1.2.1-py2.py3-none-any.whl (24 kB)
Downloading Markdown-3.7-py3-none-any.whl (106 kB)
Downloading tensorboard_data_server-0.7.2-py3-none-manylinux_2_31_x86_64.whl (6.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.6/6.6 MB 24.5 MB/s eta 0:00:00
Downloading werkzeug-3.1.3-py3-none-any.whl (224 kB)
Downloading fsspec-2024.10.0-py3-none-any.whl (179 kB)
Downloading importlib_resources-6.4.5-py3-none-any.whl (36 kB)
Downloading zipp-3.21.0-py3-none-any.whl (9.6 kB)
Downloading cachetools-5.5.0-py3-none-any.whl (9.5 kB)
Downloading MarkupSafe-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (20 kB)
Downloading pyasn1_modules-0.4.1-py3-none-any.whl (181 kB)
Downloading requests_oauthlib-2.0.0-py2.py3-none-any.whl (24 kB)
Downloading rsa-4.9-py3-none-any.whl (34 kB)
Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
Downloading pyasn1-0.6.1-py3-none-any.whl (83 kB)
Building wheels for collected packages: promise
  Building wheel for promise (setup.py) ... done
  Created wheel for promise: filename=promise-2.3-py3-none-any.whl size=21484 sha256=57269174e696e958ff43c26b4f52b68c04ed32a9e9d194703b00dec60e607648
  Stored in directory: /root/.cache/pip/wheels/54/4e/28/3ed0e1c8a752867445bab994d2340724928aa3ab059c57c8db
Successfully built promise
Installing collected packages: pytz, libclang, flatbuffers, dm-tree, zipp, wrapt, urllib3, tzdata, typing-extensions, tqdm, toml, termcolor, tensorflow-io-gcs-filesystem, tensorflow-estimator, tensorboard-data-server, six, pyparsing, pyasn1, pyarrow, psutil, protobuf, Pillow, packaging, opt-einsum, oauthlib, numpy, MarkupSafe, markdown, keras, importlib_resources, immutabledict, idna, grpcio, gast, fsspec, etils, docstring-parser, click, charset-normalizer, certifi, cachetools, absl-py, werkzeug, tensorflow-metadata, simple-parsing, scipy, rsa, requests, python-dateutil, pydot, pyasn1-modules, promise, ml-dtypes, h5py, google-pasta, astunparse, requests-oauthlib, pandas, google-auth, google-auth-oauthlib, array-record, tensorboard, tensorflow-datasets, tensorflow, tf-keras, tensorflow-hub
Successfully installed MarkupSafe-3.0.2 Pillow-11.0.0 absl-py-2.1.0 array-record-0.5.1 astunparse-1.6.3 cachetools-5.5.0 certifi-2024.8.30 charset-normalizer-3.4.0 click-8.1.7 dm-tree-0.1.8 docstring-parser-0.16 etils-1.11.0 flatbuffers-24.3.25 fsspec-2024.10.0 gast-0.6.0 google-auth-2.36.0 google-auth-oauthlib-1.2.1 google-pasta-0.2.0 grpcio-1.68.1 h5py-3.12.1 idna-3.10 immutabledict-4.2.1 importlib_resources-6.4.5 keras-2.15.0 libclang-18.1.1 markdown-3.7 ml-dtypes-0.3.2 numpy-1.26.4 oauthlib-3.2.2 opt-einsum-3.4.0 packaging-24.2 pandas-2.2.3 promise-2.3 protobuf-3.20.3 psutil-6.1.0 pyarrow-18.1.0 pyasn1-0.6.1 pyasn1-modules-0.4.1 pydot-3.0.3 pyparsing-3.2.0 python-dateutil-2.9.0.post0 pytz-2024.2 requests-2.32.3 requests-oauthlib-2.0.0 rsa-4.9 scipy-1.14.1 simple-parsing-0.1.6 six-1.17.0 tensorboard-2.15.2 tensorboard-data-server-0.7.2 tensorflow-2.15.1 tensorflow-datasets-4.9.7 tensorflow-estimator-2.15.0 tensorflow-hub-0.16.1 tensorflow-io-gcs-filesystem-0.37.1 tensorflow-metadata-1.16.1 termcolor-2.5.0 tf-keras-2.15.1 toml-0.10.2 tqdm-4.67.1 typing-extensions-4.12.2 tzdata-2024.2 urllib3-2.2.3 werkzeug-3.1.3 wrapt-1.14.1 zipp-3.21.0

Installation complete.

Error: Valid installation of TensorFlow not found.

Python environments searched for 'tensorflow' package:
 /usr/bin/python3.10

Python exception encountered:
 Traceback (most recent call last):
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 122, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 96, in _run_hook
    module = hook()
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 120, in _hook
    return _find_and_load(name, import_)
ModuleNotFoundError: No module named 'tensorflow'

You can install TensorFlow using the install_tensorflow() function.
Execution halted

Broken rocker/ml installation:

> library(tensorflow); sessionInfo()
R version 4.4.1 (2024-06-14)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 22.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] tensorflow_2.16.0.9000

loaded via a namespace (and not attached):
 [1] compiler_4.4.1    magrittr_2.0.3    Matrix_1.7-0      cli_3.6.2        
 [5] whisker_0.4.1     base64enc_0.1-3   Rcpp_1.0.12       reticulate_1.37.0
 [9] grid_4.4.1        jsonlite_1.8.8    lifecycle_1.0.4   rlang_1.1.4      
[13] tfruns_1.5.3      png_0.1-8         lattice_0.22-6

Working (older) rocker/rstudio-based installation:

R version 4.4.1 (2024-06-14)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 22.04.4 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] tensorflow_2.16.0.9000

loaded via a namespace (and not attached):
 [1] compiler_4.4.1    magrittr_2.0.3    Matrix_1.7-0      cli_3.6.3        
 [5] whisker_0.4.1     base64enc_0.1-3   Rcpp_1.0.13       reticulate_1.40.0
 [9] grid_4.4.1        jsonlite_1.8.8    lifecycle_1.0.4   rlang_1.1.4      
[13] tfruns_1.5.3      png_0.1-8         lattice_0.22-6
sneumann commented 4 days ago

Just for completeness: I compared the environment variables, found two that sound like python/reticulate used a fresh container, removed 'em, to no avail.

unset VIRTUAL_ENV
unset RETICULATE_AUTOCONFIGURE
R -e 'install.packages("reticulate"); remotes::install_github("rstudio/tensorflow", upgrade="never"); library(tensorflow); install.packages("keras"); library(keras); install_keras(); model.sla <- keras_model_sequential(input_shape = c(NULL, 12906, 1))'
benz0li commented 3 days ago

@sneumann Cross references:

sneumann commented 3 days ago

So, while not a solution, the workaround is to 

## Does not work:
R -e 'tensorflow::as_tensor("Hello World")' ; 

## Workaround:
R -e 'reticulate::use_virtualenv("/root/.virtualenvs/r-tensorflow", required = TRUE); library(tensorflow); tensorflow::as_tensor("Hello World")'
eitsupi commented 3 days ago

I'm sorry, but I couldn't figure out what bug in this was caused by this repository. Assuming this is a bug, isn't it a bug in your installation of the R package?

sneumann commented 3 days ago

So, as I have a workaround, this is not urgent. But as installation worked on rocker/rstudio from a few months back, and does not work in current rocker/ml, it is a bit of a regression. I can't put my finger in it, but something is changing/messing up the https://rstudio.github.io/reticulate/articles/versions.html#order-of-discovery of the virtual environment from what users like me were used to. Yours, Steffen

eitsupi commented 3 days ago

Isn't that simply the difference between the environment variable VIRTUAL_ENV being already set or not, as your error message says "NOTE: Python version was forced by VIRTUAL_ENV"? Maybe related to #718. cc @cboettig @yuvipanda

By the way, to be clear, I am tired of managing images with Python set up. I believe that users should be responsible for installing the version of Python and Python packages they want.