search

theislab / single-cell-tutorial

Single cell current best practices tutorial case study for the paper:Luecken and Theis, "Current best practices in single-cell RNA-seq analysis: a tutorial"
1.38k stars 457 forks source link

R libraries not accessible #24

Closed mumichae closed 3 years ago

mumichae commented 5 years ago

When using R in anaconda, a warning that the libraries cannot be installed occurs. These libraries can be added only locally. In order to overcome this, it helps to change the R_LIB path in the activate script

mumichae commented 5 years ago 
cd $CONDA_PREFIX
conda deactivate
vim etc/conda/activate.d/env_vars.sh

add

R_LIBS_OLD=$R_LIBS
export R_LIBS_OLD
export R_LIBS=""

and to etc/conda/deactivate.d/env_vars.sh add:

R_LIBS=$R_LIBS_OLD
export R_LIBS
unset R_LIBS_OLD
LuckyMD commented 5 years ago

So you want to unset the R_LIBS env variable in the environment. I didn't have any issues with this on my side previously, but will test. Thanks for flagging!

mumichae commented 5 years ago

Yes, we only had this problem when working on the interactive server

LuckyMD commented 5 years ago

thanks for letting me know! Will check it there.

giovp commented 5 years ago

Same here, on Fedora 28 also had this issue, fixed it with @mumichae suggestion. After that, also had the classic libicui18n.so.64: cannot open shared object file: No such file or directory and surprisingly fixed it with conda install -c r r-stringi plus a bunch of missing libraries.

giovp commented 5 years ago

Update, on the same Fedora 28, from rpy2.robjects import pandas2ri was throwing errors such as 

R[write to console]: Error: package or namespace load failed for ‘tools’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/tools.so':
  /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/tools.so: undefined symbol: R_NewPreciousMSet

R[write to console]: Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/tools.so':
  /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/tools.so: undefined symbol: R_NewPreciousMSet
Calls: <Anonymous> -> loadNamespace -> library.dynam -> dyn.load

This is also a classic dynamic loading problem when working on servers. Workaround was looking where tools.so was pointing: R CMD ldd /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/tools.so

and exporting the path, with: export LD_LIBRARY_PATH=/home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/:${LD_LIBRARY_PATH} . Again, this is most likely different for other users, just leaving it here for reference. Doesn't feel like a stable solution but it works...

LuckyMD commented 5 years ago

Thanks for further updates @giovp. It would be good to know which environment variables we might still be missing that are reset when moving to our new env. R_LIBS seems to be one... but I fail to see what you concluded for tool.so just $HOME/miniconda3/envs/$CONDA_PREFIX/lib/?

giovp commented 5 years ago

R CMD ldd /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/tools.so gives:

linux-vdso.so.1 (0x00007fff7cdb2000)
    libR.so => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/libR.so (0x00007faf60e25000)
    libc.so.6 => /lib64/libc.so.6 (0x00007faf60a67000)
    libRblas.so => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/./libRblas.so (0x00007faf5ee10000)
    libm.so.6 => /lib64/libm.so.6 (0x00007faf5ea7c000)
    libgomp.so.1 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/../../libgomp.so.1 (0x00007faf61488000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007faf5e85d000)
    libreadline.so.7 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/../../libreadline.so.7 (0x00007faf61437000)
    libpcre.so.1 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/../../libpcre.so.1 (0x00007faf613ef000)
    liblzma.so.5 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/../../liblzma.so.5 (0x00007faf613c6000)
    libbz2.so.1.0 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/../../libbz2.so.1.0 (0x00007faf613b2000)
    libz.so.1 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/../../libz.so.1 (0x00007faf61396000)
    librt.so.1 => /lib64/librt.so.1 (0x00007faf5e655000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007faf5e451000)
    libicuuc.so.58 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/../../libicuuc.so.58 (0x00007faf5e29e000)
    libicui18n.so.58 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/../../libicui18n.so.58 (0x00007faf5e01f000)
    /lib64/ld-linux-x86-64.so.2 (0x00007faf612c1000)
    libgfortran.so.4 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/./../../libgfortran.so.4 (0x00007faf5def1000)
    libtinfo.so.6 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/../.././libtinfo.so.6 (0x00007faf61357000)
    libicudata.so.58 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/../.././libicudata.so.58 (0x00007faf5c5ef000)
    libstdc++.so.6 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/../.././libstdc++.so.6 (0x00007faf5c47b000)
    libgcc_s.so.1 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/../.././libgcc_s.so.1 (0x00007faf61343000)
    libquadmath.so.0 => /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/./../../libquadmath.so.0 (0x00007faf61307000)

Then, I added the missing path to LD_LIBRARY_PATH, with:

export LD_LIBRARY_PATH=/home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/:${LD_LIBRARY_PATH}

These sorts of errors happened to me often when working with Rstudio server in the past.

LuckyMD commented 4 years ago

You could probably just do export LD_LIBRARY_PATH=/home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/lib/:${LD_LIBRARY_PATH} to cut out the going back and forward in the directory, no?

I would have thought that a conda env would set up the environment variables correctly to link to the relevant libs, especially as they were installed via conda in the first place. Strange that this doesn't work...

wangmhan commented 4 years ago

Hi,

I'm using macOS and have the same problem. But seems not work when used the code as mentioned about to change etc/conda/activate.d/env_vars.sh

The code I used: cd /Users/***/opt/anaconda3/envs/rpy conda deactivate vim etc/conda/activate.d/env_vars.sh add: R_LIBS_OLD=/Users/menwan/opt/anaconda3/envs/rpy/lib/R/library export R_LIBS_OLD export R_LIBS=""

R_LIBS=/Library/Frameworks/R.framework/Versions/3.5/Resources/library export R_LIBS unset R_LIBS_OLD

try to see if it work: source activate rpy R library(scran) Loading required package: BiocParallel Error: package or namespace load failed for ‘BiocParallel’ in library.dynam(lib, package, package.lib): shared object ‘BiocParallel.dylib’ not found Error: package ‘BiocParallel’ could not be loaded

Seems still use the library from rpy but not from the one previous, which the scran could be used if start from Rstudio, set up in the computer.

Is there any solutions for it?

Thanks a lot!

giovp commented 4 years ago

@LuckyMD that would probably work, will try out as soon as I have time. @wangmhan would be useful to share the error message for debugging purposes

mrichter23 commented 4 years ago

Hi, I got the same error as Giovanni:

  unable to load shared object '/usr/lib/R/library/tools/libs/tools.so':
  /usr/lib/R/library/tools/libs/tools.so: undefined symbol: R_NewPreciousMSet
Calls: <Anonymous> -> loadNamespace -> library.dynam -> dyn.load

My output from R CMD ldd /usr/lib/R/library/tools/libs/tools.so :

    libR.so => /usr/lib/R/lib/libR.so (0x00007f15e6137000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f15e5d46000)
    libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3 (0x00007f15e5ad9000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f15e573b000)
    libreadline.so.7 => /lib/x86_64-linux-gnu/libreadline.so.7 (0x00007f15e54f2000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f15e5280000)
    liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f15e505a000)
    libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f15e4e4a000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f15e4c2d000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f15e4a25000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f15e4821000)
    libicuuc.so.60 => /usr/lib/x86_64-linux-gnu/libicuuc.so.60 (0x00007f15e446a000)
    libicui18n.so.60 => /usr/lib/x86_64-linux-gnu/libicui18n.so.60 (0x00007f15e3fc9000)
    libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f15e3d9a000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f15e3b7b000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f15e69d1000)
    libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f15e3951000)
    libicudata.so.60 => /usr/lib/x86_64-linux-gnu/libicudata.so.60 (0x00007f15e1da8000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f15e1a1f000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f15e1807000)

So I tried export LD_LIBRARY_PATH=/usr/lib/R/lib/:${LD_LIBRARY_PATH} But this didn't solve the problem... I'm a little lost with that, so any input/help on that would greatly be appreciated :cookie:

giovp commented 4 years ago

Try this: R CMD ldd /usr/lib/R/library/tools/libs/tools.so and then export whatever you find there

Like this:

This is also a classic dynamic loading problem when working on servers. Workaround was looking where tools.so was pointing: R CMD ldd /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/tools.so

and exporting the path, with: export LD_LIBRARY_PATH=/home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/:${LD_LIBRARY_PATH}.

Also, I would strongly recommend you to make sure you installed all the R libraries in the specific conda environment where you have also R (an the rest of the python libraries).

I see that now files are pointing to different folders already: e.g. /lib/x86_64-linux-gnu /usr/lib and this will for sure create trouble in the future, like 100% they will.

Always do .libPaths() everytime you start an R session and make sure that all the libraries are always installed in the same path. You can modify the default R paths like this and , in my experience, you will have to modify it (if you want at all) at every R session (cause for some reasons R in conda always fall back to standard libPaths, not sure why).

mrichter23 commented 4 years ago

Thanks for the quick response. I had already done R CMD ldd /usr/lib/R/library/tools/libs/tools.so and exported what was there as I described before (I had tried both paths to be sure but only reported the one that was more likely to potentially fix the issue) Update: It seems my environment R installation was corrupted:

appears to be corrupted. The path 'lib/R/doc/html/packages.html'
has an incorrect size.
  reported size: 2946 bytes
  actual size: 28743 bytes

I fixed this by removing the environment and doing a conda clean --packages --tarballs before reinstalling the environment. However, this still didn't fix the issue and even though the R in my environment was always just pointing at one path to load libraries:

[1] "/home/maria/miniconda3/envs/hep2/lib/R/library"
>

It kept insisting to use tools.so from the /usr/lib path no matter what.

In the end installing simplegeneric and tzlocal in my base environment solved the issue, but I still find this behaviour weird. Especially because all my packages are nicely installed in my specific environment and I don't have problems importing them now that I got rpy2 to work.

Btw, your files are also pointing to different directories and I think this is a normal R-thing in this context: /home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/library/tools/libs/../../../lib/ and /lib64/

genecell commented 4 years ago

As @LuckyMD mentioned:

You could probably just do export LD_LIBRARY_PATH=/home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/lib/:${LD_LIBRARY_PATH} to cut out the going back and forward in the directory, no?

I would have thought that a conda env would set up the environment variables correctly to link to the relevant libs, especially as they were installed via conda in the first place. Strange that this doesn't work...

This directly solved the problem, cool and elegant!

Besides, stackoverflow offers the same answer rpy2 complaining with undefined symbol: Rf_translateCharUTF8:

... I think that the problem was with the system picking the wrong libR.so.

wangmhan commented 4 years ago

I finally found that as I installed R first. Then, just don't install R again through conda, the anaconda could find the existed R automatically. That's my solution, but I don't know if could fit others or not.

On Sun, 10 May 2020 at 17:21, Min Dai notifications@github.com wrote:

You could probably just do export LD_LIBRARY_PATH=/home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/lib/:${LD_LIBRARY_PATH} to cut out the going back and forward in the directory, no?

I would have thought that a conda env would set up the environment variables correctly to link to the relevant libs, especially as they were installed via conda in the first place. Strange that this doesn't work...

This directly solved the problem, cool and elegant!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/theislab/single-cell-tutorial/issues/24#issuecomment-626344061, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJVYTVX3DNPO62IG4BKGPNTRQ3A55ANCNFSM4ILUDIEA .

Manikgarg commented 4 years ago

As @LuckyMD mentioned:

You could probably just do export LD_LIBRARY_PATH=/home/icb/giovanni.palla/miniconda3/envs/sc/lib/R/lib/:${LD_LIBRARY_PATH} to cut out the going back and forward in the directory, no? I would have thought that a conda env would set up the environment variables correctly to link to the relevant libs, especially as they were installed via conda in the first place. Strange that this doesn't work...

This directly solved the problem, cool and elegant!

This solved the issue for me. Thanks a lot!