search

rocker-org / ml

experimental machine learning container
GNU General Public License v2.0
50 stars 13 forks source link

Switching between python virtualevs (e.g. support greta / multiple tensorflow versions) #29

Open cboettig opened 4 years ago

cboettig commented 4 years ago

We had set the env var RETICULATE_PYTHON_ENV in the global environmental variable so that any user will have access to a system-wide python virtual environment.

Sometimes users will want a separate virtual environment, e.g. to use packages that require older / specific versions of tensorflow, such as greta (v0.3.1). 

## set up required greta (0.3.1) env
library(reticulate)
virtualenv_install(envname="/opt/venv/greta", 
  packages = c("tensorflow==1.14.0", 
                         "tensorflow_probability==0.7.0") )

use_virtualenv("/opt/venv/greta")

## test this is 1.14.0 and not 2.0:
tensorflow::tf_version()

Unfortunately, this doesn't change the virtualenv, because it's been locked by the env var, requiring the user to override that setting, e..g by using a local .Renviron that sets RETICULATE_PYTHON_ENV=/opt/venv/greta.

A better solution is for us to set WORKON_HOME env var (e.g. to /opt/venv). This allows reticulate() to "discover" the default reticulate virtualenv we create in /opt/venv/reticulate, so we have an out-of-the-box environment, while the above commands can be run from a fresh R session to set up a new environment without having to restart the R session and mess with Renviron.

Note: when we don't force RETICULATE_PYTHON_ENV, reticulate;:py_config() harasses us to install miniconda instead of using the system Python:

reticulate::py_config()
No non-system installation of Python could be found.
Would you like to download and install Miniconda?

What does reticulate have against the system-installation of python? I get that reticulate() emphasizes user-level $HOME installs of everything over system-wide defaults, but this message seems a little too pushy given that it's already detected a perfectly viable system installation...

cboettig commented 4 years ago

p.s. note that we still have to restart our R session to switch between virtualenvs, I'm not sure why that's the case. It seems the first call to py_config() or tf_version() etc loads and locks us in to that version for the remainder of the session, even though a subsequent call to use_virtualenv() doesn't throw any message/error.