search

rstudio / tensorflow

TensorFlow for R
https://tensorflow.rstudio.com
Apache License 2.0
1.33k stars 321 forks source link

Cannot seem to install TF versions less then 2.16. #602

Open njtierney opened 3 months ago

njtierney commented 3 months ago

We require to use TF version < 2.16.0 and below, since they use keras 3, which causes breaking changes.

However when trying to install TF versions below 2.16, I cannot, I get this message:

Could not find a version that satisfies the requirement tensorflow==2.15.0 (from versions: 2.16.0rc0, 2.16.1, 2.16.2, 2.17.0rc0, 2.17.0rc1, 2.17.0)

Which makes it sound like there's some issue with which version of TF is being grabbed?

tensorflow::install_tensorflow(version = "2.15.0")
#> Virtual environment 'r-tensorflow' removed.
#> Using Python: /Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12
#> Creating virtual environment 'r-tensorflow' ...
#> + /Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12 -m venv /Users/nick/.virtualenvs/r-tensorflow
#> Done!
#> Installing packages: pip, wheel, setuptools
#> + /Users/nick/.virtualenvs/r-tensorflow/bin/python -m pip install --upgrade pip wheel setuptools
#> Virtual environment 'r-tensorflow' successfully created.
#> Using virtual environment 'r-tensorflow' ...
#> + /Users/nick/.virtualenvs/r-tensorflow/bin/python -m pip install --upgrade --no-user 'tensorflow==2.15.0.*'
#> Error: Error installing package(s): "'tensorflow==2.15.0.*'"

Created on 2024-08-05 with reprex v2.1.1

Session info ``` r sessioninfo::session_info() #> ─ Session info ─────────────────────────────────────────────────────────────── #> setting value #> version R version 4.4.1 Patched (2024-07-08 r86915) #> os macOS Sonoma 14.5 #> system aarch64, darwin20 #> ui X11 #> language (EN) #> collate en_US.UTF-8 #> ctype en_US.UTF-8 #> tz Australia/Brisbane #> date 2024-08-05 #> pandoc 3.2.1 @ /opt/homebrew/bin/ (via rmarkdown) #> #> ─ Packages ─────────────────────────────────────────────────────────────────── #> package * version date (UTC) lib source #> base64enc 0.1-3 2015-07-28 [1] CRAN (R 4.4.0) #> cli 3.6.3 2024-06-21 [1] CRAN (R 4.4.0) #> digest 0.6.36 2024-06-23 [1] CRAN (R 4.4.0) #> evaluate 0.24.0 2024-06-10 [1] CRAN (R 4.4.0) #> fastmap 1.2.0 2024-05-15 [1] CRAN (R 4.4.0) #> fs 1.6.4.9000 2024-06-26 [1] Github (r-lib/fs@714990b) #> glue 1.7.0 2024-01-09 [1] CRAN (R 4.4.0) #> here 1.0.1 2020-12-13 [1] CRAN (R 4.4.0) #> htmltools 0.5.8.1 2024-04-04 [1] CRAN (R 4.4.0) #> jsonlite 1.8.8 2023-12-04 [1] CRAN (R 4.4.0) #> knitr 1.48 2024-07-07 [1] CRAN (R 4.4.0) #> lattice 0.22-6 2024-03-20 [2] CRAN (R 4.4.1) #> lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.4.0) #> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.4.0) #> Matrix 1.7-0 2024-04-26 [2] CRAN (R 4.4.1) #> png 0.1-8 2022-11-29 [1] CRAN (R 4.4.0) #> rappdirs 0.3.3 2021-01-31 [1] CRAN (R 4.4.0) #> Rcpp 1.0.13 2024-07-17 [1] CRAN (R 4.4.0) #> reprex 2.1.1 2024-07-06 [1] CRAN (R 4.4.0) #> reticulate 1.38.0 2024-06-19 [1] CRAN (R 4.4.0) #> rlang 1.1.4 2024-06-04 [1] CRAN (R 4.4.0) #> rmarkdown 2.27 2024-05-17 [1] CRAN (R 4.4.0) #> rprojroot 2.0.4 2023-11-05 [1] CRAN (R 4.4.0) #> rstudioapi 0.16.0 2024-03-24 [1] CRAN (R 4.4.0) #> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.4.0) #> tensorflow 2.16.0 2024-04-15 [1] CRAN (R 4.4.0) #> tfruns 1.5.3 2024-04-19 [1] CRAN (R 4.4.0) #> whisker 0.4.1 2022-12-05 [1] CRAN (R 4.4.0) #> withr 3.0.1 2024-07-31 [1] CRAN (R 4.4.0) #> xfun 0.46 2024-07-18 [1] CRAN (R 4.4.0) #> yaml 2.3.10 2024-07-26 [1] CRAN (R 4.4.0) #> #> [1] /Users/nick/Library/R/arm64/4.4/library #> [2] /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library #> #> ────────────────────────────────────────────────────────────────────────────── ```

Some extra text appears when running this locally not via reprex, which is:

+ /Users/nick/.virtualenvs/r-tensorflow/bin/python -m pip install --upgrade --no-user 'tensorflow==2.15.0.*'
ERROR: Could not find a version that satisfies the requirement tensorflow==2.15.0.* (from versions: 2.16.0rc0, 2.16.1, 2.16.2, 2.17.0rc0, 2.17.0rc1, 2.17.0)
ERROR: No matching distribution found for tensorflow==2.15.0.*
Error: Error installing package(s): "'tensorflow==2.15.0.*'"

Is it possible that there is a new set of default installation versions? I get a similar message with 2.13:

tensorflow::install_tensorflow(version = "2.13.0")
#> Virtual environment 'r-tensorflow' removed.
#> Using Python: /Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12
#> Creating virtual environment 'r-tensorflow' ...
#> + /Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12 -m venv /Users/nick/.virtualenvs/r-tensorflow
#> Done!
#> Installing packages: pip, wheel, setuptools
#> + /Users/nick/.virtualenvs/r-tensorflow/bin/python -m pip install --upgrade pip wheel setuptools
#> Virtual environment 'r-tensorflow' successfully created.
#> Using virtual environment 'r-tensorflow' ...
#> + /Users/nick/.virtualenvs/r-tensorflow/bin/python -m pip install --upgrade --no-user 'tensorflow==2.13.0.*'
#> Error: Error installing package(s): "'tensorflow==2.13.0.*'"

Created on 2024-08-05 with reprex v2.1.1

Session info ``` r sessioninfo::session_info() #> ─ Session info ─────────────────────────────────────────────────────────────── #> setting value #> version R version 4.4.1 Patched (2024-07-08 r86915) #> os macOS Sonoma 14.5 #> system aarch64, darwin20 #> ui X11 #> language (EN) #> collate en_US.UTF-8 #> ctype en_US.UTF-8 #> tz Australia/Brisbane #> date 2024-08-05 #> pandoc 3.2.1 @ /opt/homebrew/bin/ (via rmarkdown) #> #> ─ Packages ─────────────────────────────────────────────────────────────────── #> package * version date (UTC) lib source #> base64enc 0.1-3 2015-07-28 [1] CRAN (R 4.4.0) #> cli 3.6.3 2024-06-21 [1] CRAN (R 4.4.0) #> digest 0.6.36 2024-06-23 [1] CRAN (R 4.4.0) #> evaluate 0.24.0 2024-06-10 [1] CRAN (R 4.4.0) #> fastmap 1.2.0 2024-05-15 [1] CRAN (R 4.4.0) #> fs 1.6.4.9000 2024-06-26 [1] Github (r-lib/fs@714990b) #> glue 1.7.0 2024-01-09 [1] CRAN (R 4.4.0) #> here 1.0.1 2020-12-13 [1] CRAN (R 4.4.0) #> htmltools 0.5.8.1 2024-04-04 [1] CRAN (R 4.4.0) #> jsonlite 1.8.8 2023-12-04 [1] CRAN (R 4.4.0) #> knitr 1.48 2024-07-07 [1] CRAN (R 4.4.0) #> lattice 0.22-6 2024-03-20 [2] CRAN (R 4.4.1) #> lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.4.0) #> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.4.0) #> Matrix 1.7-0 2024-04-26 [2] CRAN (R 4.4.1) #> png 0.1-8 2022-11-29 [1] CRAN (R 4.4.0) #> rappdirs 0.3.3 2021-01-31 [1] CRAN (R 4.4.0) #> Rcpp 1.0.13 2024-07-17 [1] CRAN (R 4.4.0) #> reprex 2.1.1 2024-07-06 [1] CRAN (R 4.4.0) #> reticulate 1.38.0 2024-06-19 [1] CRAN (R 4.4.0) #> rlang 1.1.4 2024-06-04 [1] CRAN (R 4.4.0) #> rmarkdown 2.27 2024-05-17 [1] CRAN (R 4.4.0) #> rprojroot 2.0.4 2023-11-05 [1] CRAN (R 4.4.0) #> rstudioapi 0.16.0 2024-03-24 [1] CRAN (R 4.4.0) #> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.4.0) #> tensorflow 2.16.0 2024-04-15 [1] CRAN (R 4.4.0) #> tfruns 1.5.3 2024-04-19 [1] CRAN (R 4.4.0) #> whisker 0.4.1 2022-12-05 [1] CRAN (R 4.4.0) #> withr 3.0.1 2024-07-31 [1] CRAN (R 4.4.0) #> xfun 0.46 2024-07-18 [1] CRAN (R 4.4.0) #> yaml 2.3.10 2024-07-26 [1] CRAN (R 4.4.0) #> #> [1] /Users/nick/Library/R/arm64/4.4/library #> [2] /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library #> #> ────────────────────────────────────────────────────────────────────────────── ```

And

+ /Users/nick/.virtualenvs/r-tensorflow/bin/python -m pip install --upgrade --no-user 'tensorflow==2.13.0.*'
ERROR: Could not find a version that satisfies the requirement tensorflow==2.13.0.* (from versions: 2.16.0rc0, 2.16.1, 2.16.2, 2.17.0rc0, 2.17.0rc1, 2.17.0)
ERROR: No matching distribution found for tensorflow==2.13.0.*
Error: Error installing package(s): "'tensorflow==2.13.0.*'"

Trying out reticulate::py_install() yielded similar results

reticulate::py_install(
  packages = 'tensorflow==2.15.0',
  envname = "greta-env-tf2",
  pip = TRUE
)
#> Using virtual environment 'greta-env-tf2' ...
#> + /Users/nick/.virtualenvs/greta-env-tf2/bin/python -m pip install --upgrade --no-user 'tensorflow==2.15.0'
#> Error: Error installing package(s): "'tensorflow==2.15.0'"

Created on 2024-08-05 with reprex v2.1.1

Session info ``` r sessioninfo::session_info() #> ─ Session info ─────────────────────────────────────────────────────────────── #> setting value #> version R version 4.4.1 Patched (2024-07-08 r86915) #> os macOS Sonoma 14.5 #> system aarch64, darwin20 #> ui X11 #> language (EN) #> collate en_US.UTF-8 #> ctype en_US.UTF-8 #> tz Australia/Brisbane #> date 2024-08-05 #> pandoc 3.2.1 @ /opt/homebrew/bin/ (via rmarkdown) #> #> ─ Packages ─────────────────────────────────────────────────────────────────── #> package * version date (UTC) lib source #> cli 3.6.3 2024-06-21 [1] CRAN (R 4.4.0) #> digest 0.6.36 2024-06-23 [1] CRAN (R 4.4.0) #> evaluate 0.24.0 2024-06-10 [1] CRAN (R 4.4.0) #> fastmap 1.2.0 2024-05-15 [1] CRAN (R 4.4.0) #> fs 1.6.4.9000 2024-06-26 [1] Github (r-lib/fs@714990b) #> glue 1.7.0 2024-01-09 [1] CRAN (R 4.4.0) #> htmltools 0.5.8.1 2024-04-04 [1] CRAN (R 4.4.0) #> jsonlite 1.8.8 2023-12-04 [1] CRAN (R 4.4.0) #> knitr 1.48 2024-07-07 [1] CRAN (R 4.4.0) #> lattice 0.22-6 2024-03-20 [2] CRAN (R 4.4.1) #> lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.4.0) #> Matrix 1.7-0 2024-04-26 [2] CRAN (R 4.4.1) #> png 0.1-8 2022-11-29 [1] CRAN (R 4.4.0) #> Rcpp 1.0.13 2024-07-17 [1] CRAN (R 4.4.0) #> reprex 2.1.1 2024-07-06 [1] CRAN (R 4.4.0) #> reticulate 1.38.0 2024-06-19 [1] CRAN (R 4.4.0) #> rlang 1.1.4 2024-06-04 [1] CRAN (R 4.4.0) #> rmarkdown 2.27 2024-05-17 [1] CRAN (R 4.4.0) #> rstudioapi 0.16.0 2024-03-24 [1] CRAN (R 4.4.0) #> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.4.0) #> withr 3.0.1 2024-07-31 [1] CRAN (R 4.4.0) #> xfun 0.46 2024-07-18 [1] CRAN (R 4.4.0) #> yaml 2.3.10 2024-07-26 [1] CRAN (R 4.4.0) #> #> [1] /Users/nick/Library/R/arm64/4.4/library #> [2] /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library #> #> ────────────────────────────────────────────────────────────────────────────── ``` 
+ /Users/nick/.virtualenvs/greta-env-tf2/bin/python -m pip install --upgrade --no-user 'tensorflow==2.11.0'
ERROR: Could not find a version that satisfies the requirement tensorflow==2.11.0 (from versions: 2.16.0rc0, 2.16.1, 2.16.2, 2.17.0rc0, 2.17.0rc1, 2.17.0)
ERROR: No matching distribution found for tensorflow==2.11.0
Error: Error installing package(s): "'tensorflow==2.11.0'"
t-kalinowski commented 3 months ago

Hi, thanks for opening the issue.

Two notes: First, it's possible to use keras 2 (i.e., library(keras) with TF 2.16. To make that work presently, you can install legacy keras as tf_keras, and then make sure an env var is set when keras is first loaded.

tensorflow::install_tensorflow(
   version = "release" # or "2.16" or "2.17" 
   envname = "r-tensorflow", 
   extra_packages = "tf_keras", # legacy keras
   python_version = "3.10"
)
# New R session
Sys.setenv("TF_USE_LEGACY_KERAS"="1")
tensorflow::tf_config() # initialize python, load tensorflow+keras
keras:::keras_version() < "3.0" # confirm legacy Keras in use

Second, the error you're seeing about Could not find a version that satisfies the requirement ... is because the Python version is too new for the older version of TF you're attempting to install. You can to specify an older version of Python that matches what you need via the python_version argument, like in the example above.

Note, you can also supply to python_version a string with a comma separated list of version constraints, like ">=3.9,<=3.11". You can check which Python versions work with which TensorFlow versions by consulting this table: https://www.tensorflow.org/install/source#tested_build_configurations

If you need to install a version, I recommend using reticulate::install_python(). e.g.,:

reticulate::install_python("3.10")

If you're frequently installing on linux, you might also be interested to know there are pre-built python binaries available here: https://github.com/rstudio/python-builds.

njtierney commented 2 months ago

Hi @t-kalinowski !

Thank you so much for your help here, I really appreciate it!

I had no idea that Could not find a version that satisfies the requirement ... meant that the python version was not correct - this is a huge help for debugging.

Note, you can also supply to python_version a string with a comma separated list of version constraints, like ">=3.9,<=3.11". You can check which Python versions work with which TensorFlow versions by consulting this table:

That sounds really useful, currently the process in greta is to use conda_create to set the python version:

reticulate::conda_create(
        envname = "greta-env-tf2",
        python_version = python_version
      )

Where python version is specified as a deps object (https://github.com/greta-dev/greta/pull/674)

So if we did:

reticulate::conda_create(
  envname = "greta-env-tf2",
  python_version = ">=3.9,<=3.11"
)

Do you know what the process is for choosing the python version? At the moment I think we are going to implement a (somewhat complex, but hopefully understandable) checking system based on the compatible versions of TF/TFP/Python where we know a valid set of each. This should hopefully then error before the installation begins, so then the user can be stopped earlier.

I did not know that we could use later Keras, that is really useful. I will open an issue on greta to keep track of how to manage that.

t-kalinowski commented 2 months ago

Just to clarify on conda_create(), it doesn't accept version constraints like virtualenv_starter() does. Conda will automatically fetch whatever Python version it needs, so specifying constraints isn't necessary.

virtualenv_starter() checks for suitable Python versions already installed on the host. If it doesn't find any, it will prompt you to install Python manually. You can see the recommendations for how to install Python that are shown in the error message here.

If you're updating greta's Python environment setup, take a look at the updated guidelines on managing Python dependencies in reticulate here. It might also be worth considering switching from Conda to virtual environments to avoid compatibility issues. Virtual environments are usually simpler and avoid binary compatibility problems that can cause segfaults.

You might find the functions keras3::install_keras() and tensorflow::install_tensorflow() useful as examples.