search

brokamp-group / s3

More efficient downloading and usage of files hosted on AWS S3
http://brokamp-group.github.io/s3/
Other
3 stars 2 forks source link

multiple copies of libomp.dylib prevents package from working #3

Closed erikarasnick closed 3 years ago

erikarasnick commented 3 years ago

I installed the s3 package, tried to run s3_get, was prompted to install miniconda and did, installed boto3 using reticulate. When I tried to then run s3_get again, my R session is aborted ("R encountered a fatal error. The session was terminated").

I tried to restart R several times and tried both s3_get and s3_get_files.

erikarasnick commented 3 years ago

forgot to say that this was just trying to run the examples for each function

erikarasnick commented 3 years ago

If I open R in terminal and try s3_get() there I get this error:

OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/ Abort trap: 6

erikarasnick commented 3 years ago

works if I set environment variable KMP_DUPLICATE_LIB_OK=TRUE

cole-brokamp commented 3 years ago

I think this is a mac specific problem that happens when two different python libraries are installed into the same miniconda environment with two different versions of KMP library. Can you try to install the python library nomkl to see if it fixes it?

reticulate::py_install("nomkl")
erikarasnick commented 3 years ago

installing nomkl did not fix it.

cole-brokamp commented 3 years ago

What does reticulate::py_discover_config() give you?

erikarasnick commented 3 years ago

python: /Users/RASV5G/Library/r-miniconda/envs/r-reticulate/bin/python libpython: /Users/RASV5G/Library/r-miniconda/envs/r-reticulate/lib/libpython3.6m.dylib pythonhome: /Users/RASV5G/Library/r-miniconda/envs/r-reticulate:/Users/RASV5G/Library/r-miniconda/envs/r-reticulate version: 3.6.11 | packaged by conda-forge | (default, Aug 5 2020, 20:19:23) [GCC Clang 10.0.1 ] numpy: /Users/RASV5G/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/numpy numpy_version: 1.19.4

cole-brokamp commented 3 years ago

OK, so it is using r-reticulate for your python environment. There must be some other python package installed in there that also links KMP library and a different version was linked when boto was installed. The environment variable workaround does work, but I'm worried about that being a viable solution for other users.

Let's try manually setting up your python environment to be one used strictly for this package in order to avoid the conflicts.

You can load the package with library(s3) without actually initializing or having a python library available. Before you call any package commands that would require the library, you can change your python environment with:

reticulate::conda_create("r-s3")
reticulate::use_condaenv("r-s3", required = TRUE)

Check to make sure that it is using "r-s3" instead of "r-reticulate" with"

reticulate::py_config()

You might have to restart R after these settings because I think reticulate uses the python environment itself to check on the python configuration. So its best to make these changes in a fresh R session.

Now load the package with library(s3) and you should get an error message saying you need to install the boto3 module. This is because we are using a different python environment. After you install, see if the examples work.

cole-brokamp commented 3 years ago

Make sure to use the latest version I just changed too. This will make a better error message popup when you load the s3 package.

erikarasnick commented 3 years ago

This works. However, it seems like after I installed the newest version of s3 that library(s3) now initializes python, instead of initializing only when calling a package command.

cole-brokamp commented 3 years ago

That's good to know.

Installing this package on a fresh container, the delay_load option didn't seem to be respected by reticulate on loading the package. I got an error right away when I loaded the package, so I added stop_if_no_boto to the onload instructions to provide a better error message on loading.

https://github.com/geomarker-io/s3/blob/eab636e6e7b6665eb902a63d84540b2c9a243828/R/zzz.R#L4-L7

cole-brokamp commented 3 years ago

Adding stop_if_no_boto() to onLoad doesn't seem to be a good idea, because it will force an error.

Seeing as this issue of multiple libomps being available is fixed by the use of custom python environment specific to the R package, I will close this issue and track the onload issue in a new one.