cannot use base conda environment anymore in development 1.16-9000 or -9001

[gsgxnet]

Up until the reticulate package version 1.16 (the version from the cran mirrors as of today), R version 4.0.2 (2020-06-22) reticulate can be forced to use the base conda environment by

reticulate::use_condaenv("anaconda3", required = TRUE)  
reticulate::py_available(initialize = TRUE)  
library(reticulate)  

NOTE: Python version was forced by use_python function
[1] TRUE
python:         /home/.../anaconda3/bin/python
libpython:      /home/.../anaconda3/lib/libpython3.7m.so
pythonhome:     /home/.../anaconda3:/home/gsd/anaconda3
version:        3.7.7 (default, May  7 2020, 21:25:33)  [GCC 7.3.0]
numpy:          /home/.../anaconda3/lib/python3.7/site-packages/numpy
numpy_version:  1.18.5

py_config()
then shows:

name
<chr>
anaconda3   
...

this is way or any other is not an option with the development version 1.16 - 9000 or -9001 of reticulate.

devtools::install_github('rstudio/reticulate')
Downloading GitHub repo rstudio/reticulate@HEAD
✓  checking for file ‘/tmp/RtmpRbX5HR/remotes3ff04e9fa264/rstudio-reticulate-3496f36/DESCRIPTION’ ...
...
─  building ‘reticulate_1.16-9001.tar.gz’
...

reticulate::use_condaenv("anaconda3", required = TRUE)  

Fehler in reticulate::use_condaenv("anaconda3", required = TRUE) : Unable to locate conda environment 'anaconda3'
2.
stop("Unable to locate conda environment '", condaenv, "'.")
1.
reticulate::use_condaenv("anaconda3", required = TRUE)

reticulate::py_config()

python:         /usr/bin/python3reticulate::py_config()
reticulate::conda_list()
libpython:      /usr/lib64/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so
pythonhome:     //usr://usr
version:        3.8.5 (default, Jul 20 2020, 17:41:41) [GCC]
numpy:          /usr/lib64/python3.8/site-packages/numpy
numpy_version:  1.19.1

python versions found: 
 /usr/bin/python3
 /usr/bin/python
 /home/.../anaconda3/bin/python
 /home/.../anaconda3/envs/deeplearning2020/bin/python
 /home/.../anaconda3/envs/opensap_ml2/bin/python
 /home/.../anaconda3/envs/tfml2/bin/python

but

reticulate::conda_list()

name
<chr>
python
<chr>
deeplearning2020    /home/.../anaconda3/envs/deeplearning2020/bin/python            
...

also looks like there the base environment is no more listed/seen by conda_list().

Tried use_condaenv with argument base as well. Same.

I hope this can be fixed. Or is it a feature?

Reason for using the development version is Issue #831

[gsgxnet]

Checked myself in the sources, seems to be a feature:

conda.R

  # remove root conda environment as it shouldnnt count as environment and
  # it is not correct to install packages into it.
  conda_envs <- conda_envs[!conda_envs == normalizePath(info$root_prefix)]

I wonder why the root conda environment should not count. Reading thru the in the same source file at the beginning

#' R functions for managing Python [conda
#' environments](https://conda.io/docs/user-guide/tasks/manage-environments.html).

referenced environments description on the conda site. There base is mentioned several times but never stated that base shall not be used for package installs.

Somewhere else https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html in the conda docs there is a statement:

You don't want to put programs into your base environment, though. Create separate environments to keep your programs isolated from each other

For me that has been a suggestion only which could be ignored when you know what you do.

[kevinushey]

You can force the use of the 'base' Anaconda Python with something like:

> library(reticulate)
> use_python("~/anaconda3/bin/python", required = TRUE)
> py_config()
python:         /Users/kevinushey/opt/anaconda3/bin/python
libpython:      /Users/kevinushey/opt/anaconda3/lib/libpython3.7m.dylib
pythonhome:     /Users/kevinushey/opt/anaconda3:/Users/kevinushey/opt/anaconda3
version:        3.7.4 (default, Aug 13 2019, 15:17:50)  [Clang 4.0.1 (tags/RELEASE_401/final)]
numpy:          /Users/kevinushey/opt/anaconda3/lib/python3.7/site-packages/numpy
numpy_version:  1.17.2

NOTE: Python version was forced by use_python function

Is that a suitable workaround for your case?

[gsgxnet]

Thanks for your suggestion, Kevin.

I had considered before going that way but was very unsure if a use_python directing to an anconda python would be sufficient, because I assumed an anaconda environment is also, when used by reticulate more than a python interpreter. My impression was that only use_condaenv would do. So I refrained from trying. By the py_config() output you provide it looks like the use_python pointing to an anaconda "base" python also activates its environment. Seems it would then be a suitable workaround.

I just wonder what the differences of an use_python and an use_condaenv are in this special case.

As my own workaround I had circumvented the exclusion of the base (named "anaconda3" within reticulate) environment by git cloning the development version, commenting out the line

#conda_envs <- conda_envs[!conda_envs == normalizePath(info$root_prefix)]

and install reticulate from that modified local source. Works so far, but it is no long term solution.

In the long run I must decide whether and how I will setup an reticulate specific conda environment I assume.

[kevinushey]

use_condaenv() is really just a helper that delegates to use_python(), after figuring out where the associated Python executable for the requested environment actually lives. So functionally, they're the same.

[gsgxnet]

Thanks for this info. Good to know. Then yes I will use the use_python() function/hint to activate my base conda environment.

My personal policy is - an extra environment for each project is much to much from my point of view. My base environment is updated as often as possible so it uses most of the time the current anadonda packages. And I use that environment for most projects. Only if a project fails caused by wrong / not compatible packages in the anaconda base environment, I will setup a separate environment for that project. Might be not what is recommended on most sites. But that is my quiet protest against the package version mess I regard as one of the weakest points in the Python universe..

[gsgxnet]

There is enough information available how to cope with the new „feature“ in those cases like mine.

[cliffckerr]

I came across this issue too -- it seems like it doesn't benefit anyone to make it not work to explicitly specify "base" as the conda environment.

In my case, the best workaround was to find and use the system Python executable:

use_python(Sys.which('python'))

This works, but use_condaenv('base') would've been more transparent.

[gsgxnet]

Yeah, these python environments are an issue of continuous concern. Finally I came to a point where I had to understand why the conda base environment should be kept as a minimal base. And not used as an environment in any context. Reason - my base environment went into an unresolvable update conflict. To continue to use it, it will probably need a full reinstall from scratch. Thereafter all environments based on that have to be redone as well. No fun I assume. So a bare base and special environments for every project seem to be the better setup in the long run. This python environment fragility seems very amateurish to me and far from any pro software I would pay for. But as this is free and open source I can not complain. Would have to develop my own solution if I want it the way I prefer it. Not able to do so I have to cope with what I got. And work as much as possible not in Python but in R.

[cliffckerr]

In my particular use case, I'm running R+Reticulate via a Jupyter server on a virtual machine devoted just to this. So it doesn't make a lot of sense to force the use of a virtual environment when it's the only thing running anyway. Even if the conda base env manages to get borked, you can just recreate the VM. :)

[kevinushey]

That is roughly why we recommend using a Python environment if you plan to install or update any packages.

(Personally, I'd also recommend using virtual environments via e.g. pipenv or Poetry as well, just because "administering" a Conda installation is often challenging.)

[cliffckerr]

I agree with that as a general recommendation for the majority of users -- I'd also prefer for there to be flexibility for users who have a slightly different setup! (And, I agree, it can be done with the current API, but it's somewhat unintuitive; it seems to me that anything that's valid as conda activate X should also work as use_condaenv(X). It's even worse because use_condaenv('base') fails silently without the require=T argument.)

[gsgxnet]

I agree with that as a general recommendation for the majority of users -- I'd also prefer for there to be flexibility for users who have a slightly different setup! (And, I agree, it can be done with the current API, but it's somewhat unintuitive; it seems to me that anything that's valid as conda activate X should also work as use_condaenv(X). It's even worse because use_condaenv('base') fails silently without the require=T argument.)

I do understand, yes your special usecase does really not need anything else but a base environment, crafted for it.