search

rstudio / reticulate

R Interface to Python
https://rstudio.github.io/reticulate
Apache License 2.0
1.68k stars 327 forks source link

py_config() crashes R session in RStudio #456

Open abeburnett opened 5 years ago

abeburnett commented 5 years ago

Using RStudio 1.1.463 and reticulate 1.11.1-9000 I'm having some issues. 

library(reticulate)
py_config()

Works, and displays:

> library(reticulate)
> py_config()
python:         /usr/bin/python
libpython:      /usr/lib64/python2.7/config/libpython2.7.so
pythonhome:     /usr:/usr
version:        2.7.5 (default, Oct 30 2018, 23:45:53)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
numpy:          /usr/lib64/python2.7/site-packages/numpy
numpy_version:  1.13.3

python versions found: 
 /usr/bin/python
 /home/abe-burnett/anaconda3/bin/python
 /home/abe-burnett/anaconda3/envs/wayfinder_stage/bin/python

But, if I do...

use_condaenv("wayfinder_stage")
py_config()

Then my R session crashes. However, if I use py_discover_config()...

> use_condaenv("wayfinder_stage")
> py_discover_config()
python:         /home/abe-burnett/anaconda3/envs/wayfinder_stage/bin/python
libpython:      /home/abe-burnett/anaconda3/envs/wayfinder_stage/lib/libpython2.7.so
pythonhome:     /home/abe-burnett/anaconda3/envs/wayfinder_stage:/home/abe-burnett/anaconda3/envs/wayfinder_stage
version:        2.7.15 |Anaconda, Inc.| (default, Dec 14 2018, 19:04:19)  [GCC 7.3.0]
numpy:          /home/abe-burnett/anaconda3/envs/wayfinder_stage/lib/python2.7/site-packages/numpy
numpy_version:  1.15.4

python versions found: 
 /home/abe-burnett/anaconda3/envs/wayfinder_stage/bin/python
 /usr/bin/python
 /home/abe-burnett/anaconda3/bin/python

That clearly works.

Any idea why py_config() doesn't work after executing use_condaenv()?

HM-Builds commented 4 years ago

Hello,

I am experiencing exactly the same issue when using virtualenv and use_virtualenv(). I add the python virtual environment to RStudio using the .RProfile file

library(reticulate) use_virtualenv("/Users/HMarivate/Documents/my_second_python_project/virt_env", required = TRUE)

RStudio crashes when I use

py_config()

but does not crash when I use py_config()

py_discover_config() python:C:/Users/HMarivate/Documents/my_second_python_project/virt_env/Scripts/python.exe libpython: python38.dll pythonhome: C:/Users/HMarivate/Documents/my_second_python_project/virt_env version:3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] Architecture: 64bit numpy: [NOT FOUND] NOTE: Python version was forced by use_python function

Just adding a note to this to eliminate Conda as the cause of the problem.

nathanlane commented 2 years ago

Thanks a ton. I also have this issue still.

dipterix commented 2 years ago

UPDATES: Check a later comment on an easier fix https://github.com/rstudio/reticulate/issues/456#issuecomment-1046045432

Raise my hands too.

One troubleshoot attempt that might be helpful. I think it's because when reticulate tries to load numpy, some C/C++ libraries are not compatible with R? (like BLAS or LAPACK)

  1. I force reticulate to use my own conda environment. If the environment is bare minimal (numpy not installed), py_config() works fine:
  2. When I try to install numpy from within conda environment, R crashes. However, if I install from pip (provided by conda env as well), reticulate works fine
  3. If I try to install any packages that depend on numpy on conda, for example, h5py, R crashes
  4. If I install packages that don't depend on numpy, like django, R is fine
  5. Initially I thought this could be something with numpy, however, when I tried to install pandas from either conda or pip, R crashes

Environment with this issues:

> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Monterey 12.1

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

Environments that seem to be fine:

** Other platforms not tested

dipterix commented 2 years ago

After some digging into R manual - shared BLAS, I was able to find the solution. However, this solution is kinda below the average as it changes the BLAS that R uses.

Here are my shell commands:

Step 1: Backup R's default BLAS. On Linux systems, change dylib to so

cd "/Library/Frameworks/R.framework/Resources/lib/"
mv "libRblas.dylib" "libRblas.dylib.origin"

Step 2: In your conda environment, make sure conda is using openblas by running:

conda install "libblas=*=*openblas"

Step 3: find your conda version of BLAS/LAPACK, usually in your <conda environment>/lib/libopenblas_XXX, in my case, it's /Users/dipterix/miniconda/envs/my-env/lib/libopenblas_vortexp-r0.3.18.dylib. Symlink to your R lib:

ln -s "/Users/dipterix/miniconda/envs/my-env/lib/libopenblas_vortexp-r0.3.18.dylib" "/Library/Frameworks/R.framework/Resources/lib/libRblas.dylib"

Step 4: Test

Open R from terminal (Not RStudio, because if BLAS is not linked correctly, RStudio will crash). Run sessionInfo(). If BLAS is not linked correctly, stats package won't load, and R will raise this error. If everything works fine, you should be able to see something like this:

> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Monterey 12.1

Matrix products: default
BLAS:   /Users/dipterix/miniconda/envs/my-env/lib/libopenblas_vortexp-r0.3.18.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRlapack.dylib

Now, reticulate won't crash R

dipterix commented 2 years ago

(As of Feb/19 2022, this solution has been tested on OSX only)

Further dig into the reticulate source code

https://github.com/rstudio/reticulate/blob/3c438a5995fbb1274d52886fdc3d8a71cf1ba718/R/zzz.R#L63-L71

Looks like the devs already provided "fix" to this problem: you just need to set OPENBLAS and OMP_NUM_THREADS before loading reticulate. To do this, edit your ~/.Renviron, or you could run R command usethis::edit_r_environ(), and set these environment variables.

Please DO change OPENBLAS accordingly

OPENBLAS="<path to your conda or venv environment>/lib/libblas.dylib"
OMP_NUM_THREADS=1

Restart your R, now everything works fine.

Where to find BLAS library used by conda

If you don't know how to find BLAS libpath, activate conda/venv, run python:

import numpy
numpy.__config__.show()
# blas_info:
#     libraries = ['cblas', 'blas', 'cblas', 'blas']
#     library_dirs = ['/Users/dipterix/miniconda/envs/my-env/lib']
#     include_dirs = ['/Users/dipterix/miniconda/envs/my-env/include']
#     ...

Check inside library_dirs, look for file starting with libblas (on OSX, it should be libblas.dylib. On Linux, it should be libblas.so.

Why ?

According to conda-forge's newest knowledge base, they compile and ship their libraries using a bundled version of BLAS. By default, reticulate uses R's default BLAS or /usr/local/opt/openblas, which is different to the BLAS version used by conda. When reticulate tries to load python modules such as numpy or pandas using R's BLAS, the incompatibility between different BLAS versions will crash R. This issue often exists on OSX, but could also affect some Linux users. To resolve this issue, reticulate just need to link to conda version of BLAS.

@t-kalinowski is it possible that use_condaenv also detects and change the OPENBLAS env var?