Custom build python package and missing environment variable in reticulate-python

[matthewgson]

I’m not sure if this should be classified as a bug or a misunderstanding on my part, so feel free to close this issue if it’s not a bug.

Since it’s hard to create a reproducible example, let me describe the scenario: I built a custom C++ library, QuantLib, (not using the pip version since I need intraday calculations) on my server in ~/local/(as I don’t have sudo privileges). I installed the Python bindings (via SWIG) for this package in my reticulate environment, which was created using install_miniconda(), and is located at ~/.local/share/r-miniconda/envs/r-reticulate.

I set up the package building and binding in bash using the following environment variables. The package and everything works just as expected when running the python from the bash terminal:

export PATH=$HOME/local/bin:$PATH
export CPPFLAGS="-I$HOME/local/include"
export LDFLAGS="-L$HOME/local/lib"
export LD_LIBRARY_PATH=$HOME/local/lib:$LD_LIBRARY_PATH
export PATH=~/.local/share/r-miniconda/envs/r-reticulate/bin:$PATH

However, when I try to run Python from R using reticulate repl_python(), I get an ImportError, which seems related to the PATH or LD_LIBRARY_PATH environment variable:

import QuantLib as ql
# ImportError: libQuantLib.so.1: cannot open shared object file: No such file or directory

libQuantLib.so.1 file is located at ~/local/lib as above LD_LIBRARY_PATH specifies.

I’ve tried setting these environment variables in R via .Renviron, and also directly from Python (from R-reticulate) using:

os.environ["LD_LIBRARY_PATH"] = $HOME/local/lib:$LD_LIBRARY_PATH

But the error persists.

What could be causing this issue? Below is the full error trace from R:

>>> import QuantLib
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 122, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 96, in _run_hook
    module = hook()
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 120, in _hook
    return _find_and_load(name, import_)
  File "/home/ms/.local/share/r-miniconda/envs/r-reticulate/lib/python3.10/site-packages/QuantLib-1.35-py3.10-linux-x86_64.egg/QuantLib/__init__.py", line 19, in <module>
    from .QuantLib import *
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 122, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 96, in _run_hook
    module = hook()
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 120, in _hook
    return _find_and_load(name, import_)
  File "/home/ms/.local/share/r-miniconda/envs/r-reticulate/lib/python3.10/site-packages/QuantLib-1.35-py3.10-linux-x86_64.egg/QuantLib/QuantLib.py", line 10, in <module>
    from . import _QuantLib
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 122, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 96, in _run_hook
    module = hook()
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 120, in _hook
    return _find_and_load(name, import_)
ImportError: libQuantLib.so.1: cannot open shared object file: No such file or directory

[t-kalinowski]

Are you able to load the custom built Python package outside of R/reticulate, in a vanilla Python session?

[matthewgson]

Yes, it works when run in vanilla python- on condition that environment variables has to be defined (~/.bashrc above) in prior. Vanilla python produces the same error if not defined - and I suspect reticulate python somehow doesn't recognize those env variables defined above ways.

[t-kalinowski]

Conda environments requires some workarounds unfortunately (things are simpler with virtual environments...)

Can you try re-setting LD_LIBRARY_PATH in Python and R, after reticulate is finished initializing the Python session?

E.g.:

library(reticulate)
py_available(TRUE) # initialize Python

local({
  path_to_my_sos <- "~/local/lib"

  py_env <- import("os")$environ
  py_env$update(list("LD_LIBRARY_PATH" = sprintf(
    "%s:%s", path_to_my_sos, py_env$get("LD_LIBRARY_PATH", "")
  )))
  Sys.setenv(LD_LIBRARY_PATH = sprintf(
    "%s:%s", path_to_my_sos, Sys.getenv("LD_LIBRARY_PATH", "")
  ))
})

[matthewgson]

Thank you for getting back to me so quickly. I actually removed miniconda setting and was trying to see it would work with virtual environments. Still, it didn’t work and same error persists.

Should I try the code on a virtual environment (~/.virtualenvs/r-reticulate/) Python, or if it doesn’t bother you, could you give me an update?

On Sep 12, 2024, at 9:20 AM, Tomasz Kalinowski @.***> wrote:

Conda environments requires some workarounds unfortunately (things are simpler with virtual environments...)

Can you try re-setting LD_LIBRARY_PATH in Python and R, after reticulate is finished initializing the Python session?

E.g.:

library(reticulate) py_available(TRUE) # initialize Python

local({ path_to_my_sos <- "~/local/lib"

py_env <- import("os")$environ py_env$update(list("LD_LIBRARY_PATH" = sprintf( "%s:%s", path_to_my_sos, py_env$get("LD_LIBRARY_PATH", "") ))) Sys.setenv(LD_LIBRARY_PATH = sprintf( "%s:%s", path_to_my_sos, Sys.getenv("LD_LIBRARY_PATH", "") )) }) — Reply to this email directly, view it on GitHub https://github.com/rstudio/reticulate/issues/1666#issuecomment-2346265594, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALCXW7MK4KY57CBVLJKNGUDZWGIITAVCNFSM6AAAAABOC7Q4Y6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBWGI3DKNJZGQ. You are receiving this because you authored the thread.

[t-kalinowski]

Yes, try it either way. I suspect that the LD_LIBRARY_PATH that the R session started with is not the one that's presented to the Python session.

[matthewgson]

Thank you. I reinstalled miniconda and tried, but still seeing those errors unfortunately. Please see:

> library(reticulate)
> py_config()
python:         /home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate/bin/python
libpython:      /home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate/lib/libpython3.10.so
pythonhome:     /home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate:/home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate
version:        3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0]
numpy:          /home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate/lib/python3.10/site-packages/numpy
numpy_version:  2.1.1
> py_available(TRUE)
[1] TRUE
> local({
  path_to_my_sos <- "~/local/lib"

  py_env <- import("os")$environ
  py_env$update(list("LD_LIBRARY_PATH" = sprintf(
    "%s:%s", path_to_my_sos, py_env$get("LD_LIBRARY_PATH", "")
  )))
  Sys.setenv(LD_LIBRARY_PATH = sprintf(
    "%s:%s", path_to_my_sos, Sys.getenv("LD_LIBRARY_PATH", "")
  ))
})
> py_env
Error: object 'py_env' not found
> repl_python()
Python 3.10.14 (/home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate/bin/python)
Reticulate 1.38.0 REPL -- A Python interpreter in R.
Enter 'exit' or 'quit' to exit the REPL and return to R.
>>> import QuantLib as ql
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 122, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 96, in _run_hook

When vanilla python:

bash-4.4$ echo $LD_LIBRARY_PATH
/home/gunsu.son/local/lib:/apps/squashfuse/0.5.2/lib:/opt/slurm/lib64:/home/gunsu.son/local/lib::
bash-4.4$ which python
~/.local/share/r-miniconda/envs/r-reticulate/bin/python
bash-4.4$ python
Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.executable)
/home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate/bin/python
>>> import QuantLib as ql 
>>> # runs

Please let me know if you'd like to see any relevant information I can provide. Thank you.

[t-kalinowski]

Is the conda environment activated in the 2nd code chunk? I don't see the typical (r-reticulate) or (base) conda prefix. If you activate the conda env, does your custom built module still import successfully?

Unless you have a strong motivator, I would encourage switching to virtualenvs.

# reticulate::install_python("3.11") # or use /usr/bin/python or https://github.com/rstudio/python-builds

reticulate::virtualenv_create("r-reticulate", force = TRUE)

[matthewgson]

Yes.

(r-reticulate) bash-4.4$ which python
~/.local/share/r-miniconda/envs/r-reticulate/bin/python
(r-reticulate) bash-4.4$ python
Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import QuantLib as ql
>>> 

The Python installation through pyenv didn’t work as expected, so I decided not to use that approach. Instead, I created a virtual environment (venv) using the system Python on the server. However, my concern was that the libpath configured by py_config() points to the system Python path, rather than the venv folder.

[t-kalinowski]

Would you be able to provide a minimal example I could run locally to reproduce the error?

[matthewgson]

Sure- though it may not seem "minimal" but reproducible. I was able to reproduce the same result on my local mac.

Setup r-reticulate venv, from python installed from brew

virtualenv_create("r-reticulate", python = "/opt/homebrew/bin/python3")

On terminal

source ~/.virtualenvs/r-reticulate/bin/activate
pip install setuptools
brew install boost cmake

Install QuantLib

cd ~
curl -L -O https://github.com/lballabio/QuantLib/releases/download/v1.35/QuantLib-1.35.tar.gz
tar -xvzf QuantLib-1.35.tar.gz
cd QuantLib-1.35
mkdir build
cd build
cmake .. -DQL_HIGH_RESOLUTION_DATE=ON -DBoost_INCLUDE_DIR=/opt/homebrew/opt/boost/include -DCMAKE_INSTALL_PREFIX=$HOME/local
make -j$(sysctl -n hw.ncpu)
make install

echo 'export PATH=$HOME/local/bin:$PATH' >> ~/.zshrc
source ~/.zshrc
which quantlib-config 

Build python package binding

cd ..
curl -L -O https://github.com/lballabio/QuantLib-SWIG/releases/download/v1.35/QuantLib-SWIG-1.35.tar.gz
tar -xvzf QuantLib-SWIG-1.35.tar.gz
cd QuantLib-SWIG-1.35

autoreconf --install

./configure --disable-csharp --disable-r --disable-java --disable-java-autoload --disable-java-finalizer --disable-scala CXXFLAGS="-I/opt/homebrew/opt/boost/include -std=c++17" CPPFLAGS="-I/opt/homebrew/opt/boost/include" LDFLAGS="-L/opt/homebrew/opt/boost/lib" 
make -C Python 
make install

Package is installed:

pip list

Package    Version
---------- -------
numpy      2.1.1
pip        24.2
QuantLib   1.35
QuantLib   1.35
setuptools 75.1.0
wheel      0.44.0

Testing on vanilla python

cd ~
python
import QuantLib as ql

Testing from R

library(reticulate)
repl_python()
import QuantLib as ql

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/matthewson/Library/R/arm64/4.4/library/reticulate/python/rpytools/loader.py", line 122, in _find_and_load_hook
    return _run_hook(name, _hook)