Reticulate 1.38.0 for R 4.2.1 can install but not import numpy, ...

dditomma commented 4 months ago

Is this issue back, reticulate virtualenv will not allow numpy to pass sanity checks?

Please note I have user access to my env, so ability to change most wiring is restricted. py_install( ... "--force-reinstall" ...) worked, as advised in the prior issue, but import then fails. The numpy sanity error is at the bottom:

RuntimeError: The current Numpy installation ('/local-home/repos/project/.venv/lib/python3.9/site-packages/numpy/__init__.py') fails to pass simple sanity checks. This can be caused for example by incorrect BLAS library being linked in, or by mixing package managers (pip, conda, apt, ...). Search closed numpy issues for similar problems.

utils::sessionInfo()

> utils::sessionInfo()
R version 4.2.1 (2022-06-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS/LAPACK: /usr/lib64/libopenblasp-r0.3.3.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.12       lattice_0.22-6    png_0.1-8         fansi_1.0.6       utf8_1.2.4       
 [6] grid_4.2.1        lifecycle_1.0.4   jsonlite_1.8.8    pillar_1.9.0      rlang_1.1.4      
[11] cli_3.6.3         renv_0.16.0       rstudioapi_0.16.0 Matrix_1.4-1      vctrs_0.6.5      
[16] reticulate_1.38.0 tools_4.2.1       glue_1.7.0        compiler_4.2.1

RStudio.Version()

> RStudio.Version()
$citation

To cite RStudio in publications use:

  RStudio Team (2022). RStudio: Integrated Development Environment for R. RStudio, PBC,
  Boston, MA URL http://www.rstudio.com/.

A BibTeX entry for LaTeX users is

  @Manual{,
    title = {RStudio: Integrated Development Environment for R},
    author = {{RStudio Team}},
    organization = {RStudio, PBC},
    address = {Boston, MA},
    year = {2022},
    url = {http://www.rstudio.com/},
  }

$mode
[1] "server"

$edition
[1] "Professional"

$version
[1] ‘2022.7.2.576.12’

$long_version
[1] "2022.07.2+576.pro12"

$release_name
[1] "Spotted Wakerobin"

reticulate::py_config()

> reticulate::py_config()
python:         /local-home/repos/project/.venv/bin/python
libpython:      /opt/python/3.9.7/lib/libpython3.9.so
pythonhome:     /local-home/repos/project/.venv:/local-home/repos/project/.venv
version:        3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0]
numpy:          /local-home/repos/project/.venv/lib/python3.9/site-packages/numpy
numpy_version:  1.23.3

NOTE: Python version was forced by RETICULATE_PYTHON

import error

>>> import pandas
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/local-path/.renv_paths_cache/v5/R-4.2/x86_64-pc-linux-gnu/reticulate/1.38.0/8810e64cfe0240afe926617a854a38a4/reticulate/python/rpytools/loader.py", line 122, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/local-path/.renv_paths_cache/v5/R-4.2/x86_64-pc-linux-gnu/reticulate/1.38.0/8810e64cfe0240afe926617a854a38a4/reticulate/python/rpytools/loader.py", line 96, in _run_hook
    module = hook()
  File "/local-path/.renv_paths_cache/v5/R-4.2/x86_64-pc-linux-gnu/reticulate/1.38.0/8810e64cfe0240afe926617a854a38a4/reticulate/python/rpytools/loader.py", line 120, in _hook
    return _find_and_load(name, import_)
  File "/local-home/repos/project/.venv/lib/python3.9/site-packages/pandas/__init__.py", line 11, in <module>
    __import__(_dependency)
  File "/local-path/.renv_paths_cache/v5/R-4.2/x86_64-pc-linux-gnu/reticulate/1.38.0/8810e64cfe0240afe926617a854a38a4/reticulate/python/rpytools/loader.py", line 122, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/local-path/.renv_paths_cache/v5/R-4.2/x86_64-pc-linux-gnu/reticulate/1.38.0/8810e64cfe0240afe926617a854a38a4/reticulate/python/rpytools/loader.py", line 96, in _run_hook
    module = hook()
  File "/local-path/.renv_paths_cache/v5/R-4.2/x86_64-pc-linux-gnu/reticulate/1.38.0/8810e64cfe0240afe926617a854a38a4/reticulate/python/rpytools/loader.py", line 120, in _hook
    return _find_and_load(name, import_)
  File "/local-home/repos/project/.venv/lib/python3.9/site-packages/numpy/__init__.py", line 345, in <module>
    _sanity_check()
  File "/local-home/repos/project/.venv/lib/python3.9/site-packages/numpy/__init__.py", line 343, in _sanity_check
    raise RuntimeError(msg.format(__file__)) from None
RuntimeError: The current Numpy installation ('/local-home/repos/project/.venv/lib/python3.9/site-packages/numpy/__init__.py') fails to pass simple sanity checks. This can be caused for example by incorrect BLAS library being linked in, or by mixing package managers (pip, conda, apt, ...). Search closed numpy issues for similar problems.

t-kalinowski commented 4 months ago

I suspect (but haven't tested) this is a CentOS 7 issue, with the system BLAS version installed being too old for the latest NumPy.

Does rolling back NumPy fix the issue?

py_install("numpy<2")

If that doesn't work, would you be able to reduce this down to a minimal example I could reproduce locally?

i.e.,

library(reticulate)
virtualenv_create(force = TRUE, version = "3.9")
np_array(1:3)

which for me gives:

....
Successfully installed numpy-2.0.0
Virtual environment '~/.virtualenvs/r-reticulate' successfully created.
#> np_array(1:3)
array([1, 2, 3], dtype=int32)

dditomma commented 4 months ago

Thanks for taking a look, Tomasz! Your compact example demonstrates a problem on my end. All is well until the np_array() call, which fails on a core.multiarray error, at bottom:

Error: Required version of NumPy not available: numpy.core.multiarray failed to import

Same diagnostics as before attached as r-session-fail.log.

...
Installing packages: numpy
+ ~/.virtualenvs/r-reticulate/bin/python -m pip install --upgrade --no-user numpy
Looking in indexes: https://rstudiopm.company.com/pypi/2022-10-04/simple
Collecting numpy
  Downloading https://rstudiopm.company.com/pypi/date/packages/numpy/hash/numpy-1.23.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.1/17.1 MB 230.9 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-1.23.3
Virtual environment '~/.virtualenvs/r-reticulate' successfully created.
> np_array(1:3)
Error: Required version of NumPy not available: numpy.core.multiarray failed to import

t-kalinowski commented 4 months ago

I've put together a minimal Dockerfile starting from centos7, and installing R and Python. Everything seems to work fine. Would you be able to iterate on this Dockerfile until you can reproduce the error in it?

FROM centos:7

RUN `which sudo` yum install -y https://github.com/r-lib/rig/releases/download/latest/r-rig-latest-1.$(arch).rpm

RUN rig add release

RUN R -q -e 'install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/stable/%s/%s/%s", .Platform$pkgType, R.Version()$os, R.Version()$arch))'

RUN R -q -e 'pak::pkg_install("reticulate")'

RUN yum install -y git
RUN yum install -y gcc make patch zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-devel tk-devel libffi-devel xz-devel
RUN R -q -e 'reticulate::install_python("3.9")'

RUN R -q -e 'reticulate::virtualenv_create()'

RUN R -q -e 'reticulate::np_array(1:3)'

Run from bash with:

docker build -t reticulate-centos7 -f centos7.Dockerfile .

dditomma commented 4 months ago

I don't find docker in my env ('which' finds no docker in path). I tried in the Docker Playground and the sessions spontaneously shut down just as they install_python() - the start of those last three RUN steps (screenshots, even though the playground session should not time out after a few minutes).

I'll report to our infrastructure team and see if they can track down the conflict.

attemp-docker-playground-01 attemp-docker-playground-02

t-kalinowski commented 4 months ago

Instead of building Python from source with reticulate::install_python(), you can try installing pre-built binaries from https://github.com/rstudio/python-builds?tab=readme-ov-file#rhelcentos-linux

rstudio / reticulate