lib/R/etc/ldpaths: No such file or directory

pbordron commented 5 years ago

Issue:

I get the following error when using some conda R environment with galaxy or snakemake in HPC infrastructure:

R
$CONDA_PREFIX/lib/R/bin/R: line 238: $CONDA_PREFIX/lib/R/etc/ldpaths: No such file or directory

After tracking the issue, it appears that activating the conda environment with R runs the command R CMD javareconf that updates the file $CONDA_ENV/lib/R/etc/ldpaths.

I get this issue in two different cluster infrastructures:

The first one is a galaxy VM (CentOS 6) submitting jobs to computation nodes (CentOS 6) with SGE 6.2u4.
The second one a cluster node (CentOS 7) submitting jobs to computation nodes (CentOS 7) with SGE 8.1.9.

In both case, conda envs are stored on NFSv3 or BeeGFS shares mounted on computation nodes.

Concurrent activations like it can append during a galaxy school or with some snakemake wokflows with network latency lead to two cases:

the most common: $CONDA_PREFIX/lib/R/etc/ldpaths doesn't exists during a short gap of time and another activation appends during this gap producing this error for some jobs.
the less common: something when wrong during the previous case (an error, user CTRL+C, interblocking processes, ...) and the $CONDA_PREFIX/lib/R/etc/ldpaths doesn't exists anymore. One way to solve the issue is to reinstall the r-base package.

IMHO, I think a way to solve this issue is to replace the file only when needed, not at each environment activation.

Regards

Philippe

Environment (conda list):

``` $ conda list # Name Version Build Channel _r-mutex 1.0.0 anacondar_1 bwidget 1.9.11 1 bzip2 1.0.6 h470a237_2 conda-forge ca-certificates 2018.10.15 ha4d7672_0 conda-forge cairo 1.14.12 h276e583_5 conda-forge curl 7.63.0 h74213dd_0 conda-forge expat 2.2.5 hfc679d8_2 conda-forge fontconfig 2.13.1 h65d0f4c_0 conda-forge freetype 2.9.1 h6debe1e_4 conda-forge gettext 0.19.8.1 h5e8e0c9_1 conda-forge glib 2.56.2 h464dc38_0 conda-forge graphite2 1.3.12 hfc679d8_1 conda-forge gsl 2.2.1 h0c605f7_3 harfbuzz 1.9.0 hee26f79_1 conda-forge icu 58.2 hfc679d8_0 conda-forge jpeg 9c h470a237_1 conda-forge krb5 1.16.2 hbb41f41_0 conda-forge libcurl 7.63.0 hbdb9355_0 conda-forge libedit 3.1.20170329 haf1bffa_1 conda-forge libffi 3.2.1 hfc679d8_5 conda-forge libgcc-ng 7.2.0 hdf63c60_3 conda-forge libgfortran 3.0.0 1 conda-forge libiconv 1.15 h470a237_3 conda-forge libpng 1.6.35 ha92aebf_2 conda-forge libssh2 1.8.0 h5b517e9_2 conda-forge libstdcxx-ng 7.2.0 hdf63c60_3 conda-forge libtiff 4.0.9 he6b73bb_2 conda-forge libuuid 2.32.1 h470a237_2 conda-forge libxcb 1.13 h470a237_2 conda-forge libxml2 2.9.8 h422b904_5 conda-forge make 4.2.1 h470a237_1003 conda-forge ncurses 6.1 hfc679d8_1 conda-forge openssl 1.0.2p h470a237_1 conda-forge pango 1.40.14 he752989_2 conda-forge pcre 8.41 hfc679d8_3 conda-forge pixman 0.34.0 h470a237_3 conda-forge pthread-stubs 0.4 h470a237_1 conda-forge r 3.5.1 r351_1000 conda-forge r-assertthat 0.2.0 r351h6115d3f_1001 conda-forge r-backports 1.1.3 r351hc070d10_0 conda-forge r-base 3.5.1 h391c2eb_4 conda-forge r-boot 1.3_20 r351_1000 conda-forge r-callr 3.1.1 r351h6115d3f_1000 conda-forge r-class 7.3_15 r351hc070d10_0 conda-forge r-cli 1.0.1 r351h6115d3f_1000 conda-forge r-clipr 0.4.1 r351h6115d3f_1001 conda-forge r-clisymbols 1.2.0 r351h6115d3f_1001 conda-forge r-cluster 2.0.7_1 r351h364d78e_0 conda-forge r-codetools 0.2_16 r351h6115d3f_1000 conda-forge r-crayon 1.3.4 r351h6115d3f_1001 conda-forge r-curl 3.2 r351hc070d10_2 conda-forge r-desc 1.2.0 r351h6115d3f_1001 conda-forge r-devtools 2.0.1 r351h6115d3f_1000 conda-forge r-digest 0.6.18 r351hc070d10_0 conda-forge r-foreign 0.8_71 r351hc070d10_2 conda-forge r-fs 1.2.6 r351h9d2a408_0 conda-forge r-gh 1.0.1 r351h6115d3f_1001 conda-forge r-git2r 0.23.0 r351h1b3b946_0 conda-forge r-glue 1.3.0 r351h470a237_2 conda-forge r-httr 1.4.0 r351h6115d3f_1000 conda-forge r-ini 0.3.1 r351h6115d3f_1001 conda-forge r-jsonlite 1.6 r351hc070d10_0 conda-forge r-kernsmooth 2.23_15 r351h364d78e_2 conda-forge r-lattice 0.20_38 r351hc070d10_0 conda-forge r-magrittr 1.5 r351h6115d3f_1001 conda-forge r-mass 7.3_51.1 r351hc070d10_0 conda-forge r-matrix 1.2_15 r351hc070d10_0 conda-forge r-memoise 1.1.0 r351h6115d3f_1001 conda-forge r-mgcv 1.8_26 r351hc070d10_0 conda-forge r-mime 0.6 r351hc070d10_0 conda-forge r-nlme 3.1_137 r351h364d78e_0 conda-forge r-nnet 7.3_12 r351hc070d10_2 conda-forge r-openssl 1.1 r351h9f97512_0 conda-forge r-pkgbuild 1.0.2 r351h6115d3f_1001 conda-forge r-pkgload 1.0.2 r351h9d2a408_0 conda-forge r-prettyunits 1.0.2 r351h6115d3f_1001 conda-forge r-processx 3.2.1 r351hc070d10_0 conda-forge r-ps 1.3.0 r351hc070d10_0 conda-forge r-r6 2.3.0 r351h6115d3f_1000 conda-forge r-rcmdcheck 1.3.2 r351h6115d3f_1000 conda-forge r-rcpp 1.0.0 r351h9d2a408_0 conda-forge r-recommended 3.5.1 r351_1001 conda-forge r-remotes 2.0.2 r351h6115d3f_1000 conda-forge r-rlang 0.3.0.1 r351h470a237_0 conda-forge r-rpart 4.1_13 r351hc070d10_2 conda-forge r-rprojroot 1.3_2 r351h6115d3f_1001 conda-forge r-rstudioapi 0.8 r351h6115d3f_1001 conda-forge r-sessioninfo 1.1.1 r351h6115d3f_1000 conda-forge r-spatial 7.3_11 r351hc070d10_2 conda-forge r-survival 2.43_3 r351hc070d10_0 conda-forge r-usethis 1.4.0 r351h6115d3f_1000 conda-forge r-whisker 0.3_2 r351h6115d3f_1001 conda-forge r-withr 2.1.2 r351h6115d3f_1000 conda-forge r-xml 3.98_1.16 r351hc070d10_0 conda-forge r-xopen 1.0.0 r351h6115d3f_1000 conda-forge readline 7.0 haf1bffa_1 conda-forge tk 8.6.8 ha92aebf_0 conda-forge tktable 2.10 h14c3975_0 udunits2 2.2.25 hd30922c_1 xorg-kbproto 1.0.7 h470a237_2 conda-forge xorg-libice 1.0.9 h470a237_4 conda-forge xorg-libsm 1.2.2 h8c8a85c_6 conda-forge xorg-libx11 1.6.6 h470a237_0 conda-forge xorg-libxau 1.0.8 h470a237_6 conda-forge xorg-libxdmcp 1.1.2 h470a237_7 conda-forge xorg-libxext 1.3.3 h470a237_4 conda-forge xorg-libxrender 0.9.10 h470a237_2 conda-forge xorg-libxt 1.1.5 h470a237_2 conda-forge xorg-renderproto 0.11.1 h470a237_2 conda-forge xorg-xextproto 7.3.0 h470a237_2 conda-forge xorg-xproto 7.0.31 h470a237_7 conda-forge xz 5.2.4 h470a237_1 conda-forge zlib 1.2.11 h470a237_3 conda-forge ```

Details about conda and system ( conda info ):

``` $ conda info active environment : None user config file : $HOME/.condarc populated config files : $HOME/.condarc conda version : 4.5.12 conda-build version : 3.17.5 python version : 3.7.1.final.0 base environment : $HOME/miniconda3 (writable) channel URLs : https://conda.anaconda.org/bioconda/linux-64 https://conda.anaconda.org/bioconda/noarch https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/linux-64 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/pro/linux-64 https://repo.anaconda.com/pkgs/pro/noarch package cache : $HOME/miniconda3/pkgs $HOME/.conda/pkgs envs directories : $HOME/miniconda3/envs / $HOME/.conda/envs platform : linux-64 user-agent : conda/4.5.12 requests/2.21.0 CPython/3.7.1 Linux/3.10.0-862.14.4.el7.x86_64 centos/7 glibc/2.17 UID:GID : XXXXXXX:YYYYYYY netrc file : None offline mode : False ```

mingwandroid commented 5 years ago

IMHO, I think a way to solve this issue is to replace the file only when needed, not at each environment activation.

This is exactly when the file needs to be updated.

Can you not mount your envs on another file system? NFS is often problematic.

My only suggestion would be to try to minimize these time windows. You could copy the original ldpaths to e.g. ldpaths.$$, modify that, then replace ldpaths with ldpaths.$$. This could be coupled with a loop to make sure ldpaths exists at the point of consumption with a limit on the number of times it spins waiting for it to appear.

Please feel free to submit a PR. Personally I cannot reproduce your error since I do not use NFS.

pbordron commented 5 years ago

The point is that R CMD javareconf replaces ldpaths file each time it is called, even if there is no change (see the script here)

The best solution will be to have a ldpaths file that is proper to each conda env activation, but it is not so simple.

An easy but yet incomplete solution can be to modify the following patch such as the files are only replaced when the new file is different.

Here a proposition that is haven't tested: https://github.com/pbordron/r-base-feedstock/blob/d6aef51add58ad802f5d04467504134bf31817ac/recipe/0017-Foribly-remove-then-forcibly-mv-in-javareconf.in.patch

mingwandroid commented 5 years ago

The best solution will be to have a ldpaths file that is proper to each conda env activation, but it is not so simple.

It is not so simple at all, in fact it's impossible. There is no such thing as a 'file that is proper to each conda env' because we deliberately support a range of Javas both from conda packages, from the system package manager and 3rd party. I have gone to extensive lengths to make this work.

You've replaced a race condition around ldpaths with one around ldpaths.new, an improvement but my suggestion is still better. You can get the best of both worlds by replacing .new in your patch with .new.$$

Please work on a PR if you want this to work on nfs.

johanneskoester commented 5 years ago

@pbordron it is indeed important that conda packages work on NFS. If I can help you with the PR, please let me know. I won't have the time to work on it on my own, but I would be happy to help in any way. Thanks a lot for pointing this out!

pbordron commented 5 years ago

From now, I do not have the time nor the environment for looking for a fix. The problem does not only append with NFS, but also with BeeGFS. The fact that the ldpaths is replaced in two times is the main cause (i.e file ldpaths is removed and then ldpaths.new is moved to ldpaths).

RodrigoGM commented 5 years ago

Hello, I've recently encountered this error in my miniconda3 as well. I have a brand new miniconda3 install due to a corrupt libevent file last week. In the new install, the error

~/opt/miniconda3/bin/R: line 238: ~/opt/miniconda3/lib/R/etc/ldpaths: No such file or directory

began once I created a new environment. I also use a cluster with CentOS_6 and CentOS_7 nodes, all under LSF. Is there a any other way to fix to this issue other than the patch mentioned above ? Any other ideas on how to fix the issue ?

Many thanks in advance,

base environment ``` (base) bash-4.1$ conda list ```

``` # packages in environment at $HOME/opt/miniconda3: # # Name Version Build Channel _r-mutex 1.0.0 anacondar_1 defaults asn1crypto 0.24.0 py37_0 defaults binutils_impl_linux-64 2.31.1 h6176602_1 conda-forge binutils_linux-64 2.31.1 h6176602_3 conda-forge bowtie 1.0.0 1 bioconda bwa 0.7.17 h84994c4_4 bioconda bwidget 1.9.11 1 defaults byobu 5.98 hb42da9c_2 bioconda bzip2 1.0.6 h14c3975_1002 conda-forge ca-certificates 2018.11.29 ha4d7672_0 conda-forge cairo 1.14.12 h8948797_3 defaults certifi 2018.11.29 py37_1000 conda-forge cffi 1.11.5 py37he75722e_1 defaults chardet 3.0.4 py37_1 defaults conda 4.6.2 py37_0 conda-forge conda-env 2.6.0 1 defaults cryptography 2.4.2 py37h1ba5d50_0 defaults curl 7.62.0 hbc83047_0 defaults emacs 26.1 he9320e1_1002 conda-forge fontconfig 2.13.0 h9420a91_0 defaults freetype 2.9.1 h94bbf69_1005 conda-forge fribidi 1.0.5 h14c3975_1000 conda-forge gcc_impl_linux-64 7.3.0 habb00fd_1 conda-forge gcc_linux-64 7.3.0 h553295d_3 conda-forge gettext 0.19.8.1 h9745a5d_1001 conda-forge gfortran_impl_linux-64 7.3.0 hdf63c60_1 conda-forge gfortran_linux-64 7.3.0 h553295d_3 conda-forge giflib 5.1.4 h14c3975_1001 conda-forge glib 2.56.2 had28632_1001 conda-forge gnutls 3.6.5 hd3a4fd2_1001 conda-forge graphite2 1.3.13 hf484d3e_1000 conda-forge gsl 2.4 h14c3975_4 defaults gxx_impl_linux-64 7.3.0 hdf63c60_1 conda-forge gxx_linux-64 7.3.0 h553295d_3 conda-forge harfbuzz 1.9.0 he243708_1001 conda-forge icu 58.2 hf484d3e_1000 conda-forge idna 2.8 py37_0 defaults jpeg 9c h14c3975_1001 conda-forge krb5 1.16.1 h173b8e3_7 defaults ldc 1.11.0 hb2c9046_0 conda-forge libcurl 7.62.0 h20c2e04_0 defaults libdeflate 1.0 h14c3975_1 bioconda libedit 3.1.20170329 h6b74fdf_2 defaults libevent 2.0.22 0 defaults libffi 3.2.1 hd88cf55_4 defaults libgcc-ng 8.2.0 hdf63c60_1 defaults libgfortran-ng 7.3.0 hdf63c60_0 defaults libiconv 1.15 h14c3975_1004 conda-forge libpng 1.6.36 h84994c4_1000 conda-forge libssh2 1.8.0 1 conda-forge libstdcxx-ng 8.2.0 hdf63c60_1 defaults libtiff 4.0.10 h648cc4a_1001 conda-forge libuuid 1.0.3 1 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxml2 2.9.8 newt 0.52.20 py37h14c3975_1000 conda-forge oniguruma 6.9.0 h14c3975_1001 conda-forge openssl 1.1.1a h14c3975_1000 conda-forge pango 1.42.4 h049681c_0 defaults pcre 8.42 h439df22_0 defaults perl 5.26.2 h14c3975_1001 conda-forge pip 18.1 py37_0 defaults pixman 0.34.0 h14c3975_1003 conda-forge popt 1.16 1 bioconda pthread-stubs 0.4 h14c3975_1001 conda-forge pycosat 0.6.3 py37h14c3975_0 defaults pycparser 2.19 py37_0 defaults pyopenssl 18.0.0 py37_0 defaults pysocks 1.6.8 py37_0 defaults python 3.7.1 h0371630_7 defaults r-assertthat 0.2.0 r351h6115d3f_0 r r-backports 1.1.2 r351h96ca727_0 r r-base 3.5.1 h1e0a451_2 r r-base64enc 0.1_3 r351h96ca727_4 r r-bh 1.66.0_1 r351h6115d3f_0 r r-bindr 0.1.1 r351h6115d3f_0 r r-bindrcpp 0.2.2 r351h29659fb_0 r r-broom 0.5.0 r351h6115d3f_0 r r-callr 2.0.4 r351h6115d3f_0 r r-cellranger 1.1.0 r351h6115d3f_0 r r-cli 1.0.0 r351h6115d3f_1 r r-clipr 0.4.1 r351h6115d3f_0 r r-colorspace 1.3_2 r351h96ca727_0 r r-crayon 1.3.4 r351h6115d3f_0 r r-curl 3.2 r351hadc6856_1 r r-dbi 1.0.0 r351h6115d3f_0 r r-dbplyr 1.2.2 r351hf348343_0 r r-dichromat 2.0_0 r351h6115d3f_4 r r-digest 0.6.15 r351h96ca727_0 r r-dplyr 0.7.6 r351h29659fb_0 r r-evaluate 0.11 r351h6115d3f_0 r r-fansi 0.2.3 r351h96ca727_0 r r-forcats 0.3.0 r351h6115d3f_0 r r-ggplot2 3.0.0 r351h6115d3f_0 r r-glue 1.3.0 r351h96ca727_0 r r-gtable 0.2.0 r351h6115d3f_0 r r-haven 1.1.2 r351h29659fb_0 r r-highr 0.7 r351h6115d3f_0 r r-hms 0.4.2 r351h6115d3f_0 r r-htmltools 0.3.6 r351h29659fb_0 r r-httr 1.3.1 r351h6115d3f_1 r r-jsonlite 1.5 r351h96ca727_0 r r-knitr 1.20 r351h6115d3f_0 r r-labeling 0.3 r351h6115d3f_4 r r-lattice 0.20_35 r351h96ca727_0 r r-lazyeval 0.2.1 r351h96ca727_0 r r-lubridate 1.7.4 r351h29659fb_0 r r-magrittr 1.5 r351h6115d3f_4 r r-markdown 0.8 r351h96ca727_0 r r-mass 7.3_50 r351h96ca727_0 r r-matrix 1.2_14 r351h96ca727_0 r r-mgcv 1.8_24 r351h96ca727_0 r r-mime 0.5 r351h96ca727_0 r r-mo r-openssl 1.0.2 r351h96ca727_1 r r-pillar 1.3.0 r351h6115d3f_0 r r-pkgconfig 2.0.1 r351h6115d3f_0 r r-plogr 0.2.0 r351h6115d3f_0 r r-plyr 1.8.4 r351h29659fb_0 r r-praise 1.0.0 r351h6115d3f_4 r r-processx 3.1.0 r351h29659fb_0 r r-purrr 0.2.5 r351h96ca727_0 r r-r6 2.2.2 r351h6115d3f_0 r r-rcolorbrewer 1.1_2 r351h6115d3f_0 r r-rcpp 0.12.18 r351h29659fb_0 r r-readr 1.1.1 r351h29659fb_0 r r-readxl 1.1.0 r351h29659fb_0 r r-rematch 1.0.1 r351h6115d3f_0 r r-reprex 0.2.0 r351h6115d3f_0 r r-reshape2 1.4.3 r351h29659fb_0 r r-rlang 0.2.1 r351h96ca727_0 r r-rmarkdown 1.10 r351h6115d3f_0 r r-rprojroot 1.3_2 r351h6115d3f_0 r r-rstudioapi 0.7 r351h6115d3f_0 r r-rvest 0.3.2 r351h6115d3f_0 r r-scales 0.5.0 r351h29659fb_0 r r-selectr 0.4_1 r351h6115d3f_0 r r-stringi 1.2.4 r351h29659fb_0 r r-stringr 1.3.1 r351h6115d3f_0 r r-testthat 2.0.0 r351h29659fb_0 r r-tibble 1.4.2 r351h96ca727_0 r r-tidyr 0.8.1 r351h29659fb_0 r r-tidyselect 0.2.4 r351h29659fb_0 r r-tidyverse 1.2.1 r351h6115d3f_0 r r-tinytex 0.6 r351h6115d3f_0 r r-utf8 1.1.4 r351h96ca727_0 r r-viridislite 0.3.0 r351h6115d3f_0 r r-whisker 0.3_2 r351hf348343_4 r r-withr 2.1.2 r351h6115d3f_0 r r-xfun 0.3 r351h6115d3f_0 r r-xml2 1.2.0 r351h29659fb_0 r r-yaml 2.2.0 r351h96ca727_0 r readline 7.0 h7b6447c_5 defaults requests 2.21.0 py37_0 defaults ruamel_yaml 0.15.46 py37h14c3975_0 defaults sambamba 0.6.8 h682856c_1 bioconda samtools 1.9 h91753b0_5 bioconda setuptools 40.6.3 py37_0 defaults six 1.12.0 py37_0 defaults slang 2.3.2 hbb9399a_1000 conda-forge sqlite 3.26.0 h7b6447c_0 defaults star 2.6.1b 0 bioconda tk 8.6.8 hbc83047_0 defaults tktable 2.10 h14c3975_0 defaults tmux 2.7 h87f082e_1003 conda-forge urllib3 1.24.1 py37_0 defaults wheel 0.32.3 py37_0 defaults xorg-fixesproto 5.0 h14c3975_1002 conda-forge xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.9 h14c3975_1004 conda-forge xorg-libsm 1.2.2 h470a237_5 conda-forge xorg-libx11 1.6.7 h14c3975_1000 conda-forge xorg-libxau 1.0.8 h14c3975_1006 conda-forge xorg-libxaw 1.0.13 h14c3975_1002 conda-forge xorg-libxdmcp 1.1.2 h14c3975_1007 conda-forge xorg-libxext 1.3.3 h14c3975_1004 conda-forge xorg-libxfixes 5.0.3 h14c3975_1004 conda-forge xorg-libxft 2.3.2 h7c24a56_3 conda-forge xorg-libxmu 1.1.2 h14c3975_1002 conda-forge xorg-libxpm 3.5.12 h14c3975_1002 conda-forge xorg-libxrender 0.9.10 h14c3975_1002 conda-forge xorg-libxt 1.1.5 h14c3975_1002 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h14c3975_1002 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.4 h14c3975_4 defaults yaml 0.1.7 had09818_2 defaults zlib 1.2.11 h7b6447c_3 defaults ```

second environment ``` (base) bash-4.1$ conda list -n cnv ```

``` (base) bash-4.1$ conda list -n cnv # packages in environment at $HOME/opt/miniconda3/envs/cnv: # # Name Version Build Channel _r-mutex 1.0.0 anacondar_1 defaults binutils_impl_linux-64 2.31.1 h6176602_1 conda-forge binutils_linux-64 2.31.1 h6176602_3 conda-forge bioconductor-biobase 2.42.0 r351h14c3975_0 bioconda bioconductor-biocgenerics 0.28.0 r351_1 bioconda bioconductor-biocparallel 1.16.2 r351h1c2f66e_1 bioconda bioconductor-biostrings 2.50.1 r351h14c3975_0 bioconda bioconductor-cghbase 1.42.0 r351_0 bioconda bioconductor-cghcall 2.44.0 r351_0 bioconda bioconductor-dnacopy 1.56.0 r351h9ac9557_0 bioconda bioconductor-genomeinfodb 1.18.1 r351_0 bioconda bioconductor-genomeinfodbdata 1.2.0 r351_0 bioconda bioconductor-genomicranges 1.34.0 r351h14c3975_0 bioconda bioconductor-impute 1.56.0 r351h9ac9557_0 bioconda bioconductor-iranges 2.16.0 r351h14c3975_0 bioconda bioconductor-limma 3.38.3 r351h14c3975_0 bioconda bioconductor-marray 1.60.0 r351_0 bioconda bioconductor-qdnaseq 1.18.0 r351_0 bioconda bioconductor-rsamtools 1.34.0 r351hf484d3e_0 bioconda bioconductor-s4vectors 0.20.1 r351h14c3975_0 bioconda bioconductor-xvector 0.22.0 r351h14c3975_0 bioconda bioconductor-zlibbioc 1.28.0 r351h14c3975_0 bioconda bwidget 1.9.11 1 defaults bzip2 1.0.6 h14c3975_1002 conda-forge ca-certificates 2018.11.29 ha4d7672_0 conda-forge cairo 1.14.12 h80bd089_1005 conda-forge curl 7.63.0 h646f8bb_1000 conda-forge fontconfig 2.13.1 h2176d3f_1000 conda-forge freetype 2.9.1 h94bbf69_1005 conda-forge gcc_impl_linux-64 7.3.0 habb00fd_1 conda-forge gcc_linux-64 7.3.0 h553295d_3 conda-forge gettext 0.19.8.1 h9745a5d_1001 conda-forge gfortran_impl_linux-64 7.3.0 hdf63c60_1 conda-forge gfortran_linux-64 7.3.0 h553295d_3 conda-forge glib 2.56.2 had28632_1001 conda-forge graphite2 1.3.13 hf484d3e_1000 conda-forge gxx_impl_linux-64 7.3.0 hdf63c60_1 conda-forge gxx_linux-64 7.3.0 h553295d_3 conda-forge harfbuzz 1.9.0 he243708_1001 conda-forge icu 58.2 hf484d3e_1000 conda-forge jpeg 9c h14c3975_1001 conda-forge krb5 1.16.3 hc83ff2d_1000 conda-forge libcurl 7.63.0 h01ee5af_1000 conda-forge libedit 3.1.20170329 hf8c457e_1001 conda-forge libffi 3.2.1 hf484d3e_1005 conda-forge libgcc-ng 7.3.0 hdf63c60_0 conda-forge libgfortran-ng 7.3.0 hdf63c60_0 defaults libiconv 1.15 h14c3975_1004 conda-forge libpng 1.6.36 h84994c4_1000 conda-forge libssh2 1.8.0 h1ad7b7a_1003 conda-forge libstdcxx-ng 7.3.0 hdf63c60_0 conda-forge libtiff 4.0.10 h648cc4a_1001 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxml2 2.9.8 h143f9aa_1005 conda-forge make 4.2.1 h14c3975_2004 conda-forge ncurses 6.1 hf484d3e_1002 conda-forge openssl 1.0.2p h14c3975_1002 conda-forge pango 1.40.14 hf0c64fd_1003 conda-forge pcre 8.41 hf484d3e_1003 conda-forge pixman 0.34.0 h14c3975_1003 conda-forge pthread-stubs 0.4 h14c3975_1001 conda-forge r-base 3.5.1 he45234b_1005 conda-forge r-bh 1.66.0_1 r351h6115d3f_0 r r-bitops 1.0_6 r351h96ca727_4 r r-formatr 1.5 r351h6115d3f_0 r r-futile.logger 1.4.3 r351h6115d3f_1001 conda-forge r-futile.options 1.0.1 r351h6115d3f_1000 conda-forge r-lambda.r 1.2.3 r351h6115d3f_1000 conda-forge r-matrixstats 0.54.0 r351h96ca727_1000 conda-forge r-r.methodss3 1.7.1 r351h6115d3f_0 r r-r.oo 1.22.0 r351h6115d3f_0 r r-r.utils 2.6.0 r351h6115d3f_0 r r-rcurl 1.95_4.11 r351h96ca727_0 r r-snow 0.4_3 r351h6115d3f_1000 conda-forge r-snowfall 1.84_6.1 r351h6115d3f_1001 conda-forge readline 7.0 hf8c457e_1001 conda-forge tk 8.6.9 h84994c4_1000 conda-forge tktable 2.10 h14c3975_0 defaults wget 1.19.5 h1ad7b7a_0 defaults xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.9 h14c3975_1004 conda-forge xorg-libsm 1.2.3 h4937e3b_1000 conda-forge xorg-libx11 1.6.7 h14c3975_1000 conda-forge xorg-libxau 1.0.8 h14c3975_1006 conda-forge xorg-libxdmcp 1.1.2 h14c3975_1007 conda-forge xorg-libxext 1.3.3 h14c3975_1004 conda-forge xorg-libxrender 0.9.10 h14c3975_1002 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h14c3975_1002 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.4 h14c3975_1001 conda-forge zlib 1.2.11 h14c3975_1004 conda-forge ```

conda install info ``` (base) bash-4.1$ conda info ```

``` active environment : base active env location : $HOME/opt/miniconda3 shell level : 1 user config file : $HOME/.condarc populated config files : $HOME/.condarc conda version : 4.6.2 conda-build version : not installed python version : 3.7.1.final.0 base environment : $HOME/opt/miniconda3 (writable) channel URLs : https://conda.anaconda.org/r/linux-64 https://conda.anaconda.org/r/noarch https://conda.anaconda.org/bioconda/linux-64 https://conda.anaconda.org/bioconda/noarch https://conda.anaconda.org/dranew/linux-64 https://conda.anaconda.org/dranew/noarch https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/linux-64 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : $HOME/opt/miniconda3/pkgs $HOME/.conda/pkgs envs directories : $HOME/opt/miniconda3/envs $HOME/.conda/envs platform : linux-64 user-agent : conda/4.6.2 requests/2.21.0 CPython/3.7.1 Linux/2.6.32-696.3.2.el6.x86_64 centos/6.9 glibc/2.12 UID:GID : ...:... netrc file : $HOME/.netrc offline mode : False ```

pbordron commented 5 years ago

@RodrigoGM the patch is not tested and replace a race condition with another.

lecorguille commented 5 years ago

Hi, We're facing the same issue with our Galaxy instance.

/some_path/_conda/envs/mulled-v1-10760b69799f3df14ac223943c74682453f4c44945480a6c4ee3ccaff1510f7f/lib/R/etc/ldpaths: No such file or directory

I have to reinstall r-base within the env:

conda install r-base==3.5.1 --force-reinstall

But I don't know how many time it will be up.

So as suggested by @pbordron (we know each other), I edited the bash script

vi /some_path/_conda/envs/mulled-v1-10760b69799f3df14ac223943c74682453f4c44945480a6c4ee3ccaff1510f7f/etc/conda/activate.d/activate-r-base.sh

#!/usr/bin/env sh
#R CMD javareconf > /dev/null 2>&1 || true

It should be harmless in the context of Galaxy since we never stack env. 🤞

(FYI: @fgiacomoni)

NTLx commented 5 years ago

I also had this issue, and solved this in a tricky way.

In my other conda env, there was another ldpaths file, the path like ~/miniconda3/envs/py2.7/lib/R/etc/ldpaths, I copy this ldpaths file to where it is needed, and change the path in this file to fit my need, then everything is OK.

I may face this problem again while change conda envs, but at least I could return work ASAP. (I already backup ldpaths file for emergency)

Looking forward to someone who can give the perfect solution.

Fedorov113 commented 5 years ago

I ran into this issue with Snakemake. I use it with --use-conda flag, and when running parallel jobs with R, at some point ldpaths file just disappears. Nothing helped me to overcome this yet, really waiting for the solution.

Finesim97 commented 5 years ago

I ran into this issue with Snakemake. I use it with --use-conda flag, and when running parallel jobs with R, at some point ldpaths file just disappears. Nothing helped me to overcome this yet, really waiting for the solution.

Hey I had the exact same problem (running snakelike in parallel) and lecorguille solution helped. Inside the currently used conda directory (should be in the error log or just look for the most recent folder) inside .snakemake/conda/<hashhere> and edit the file .snakemake/conda/<hashhere>/etc/conda/activate.d/activate-r-base.sh and comment out the line described. This may not work with your setup, but "worked for me".

From

R CMD javareconf > /dev/null 2>&1 || true

to

# R CMD javareconf > /dev/null 2>&1 || true

Best Regards

Fedorov113 commented 5 years ago

@Finesim97 well, I've just commented lines from 416-432 in env/lib/R/bin/javareconf (the ones that are responsible for updating files) and everything works great. It is partially equal to your solution. Thank you!

But what's the point of updating ldpaths and Makeconf each activation anyway? I don't think users update their Java that often. Setting values for this files on first installation should be enough. And than maybe we can introduce possibility of manually invoking update? If the user reinstalled Java or something.

@mingwandroid said that activation is

exactly when the file needs to be updated.

But as it can be seen, users just comment out this step altogether.

mingwandroid commented 5 years ago

If you want to query why the R foundation made this choice you should ask them.

I don't think users update their Java that often

That you do not need this doesn't mean others do not. We need a solution that works correctly in all cases (and it's not difficult either). Reconfiguring Java for each env is not going to be undone for it is the correct thing to do. If you convince R upstream to remove javareconf form R altogether (say moving it into rJava itself) then that would also work, however I don't know enough though too say this is a good idea or even possible.

valscherz commented 5 years ago

I ran into this issue with Snakemake. I use it with --use-conda flag, and when running parallel jobs with R, at some point ldpaths file just disappears. Nothing helped me to overcome this yet, really waiting for the solution.

Hey I had the exact same problem (running snakelike in parallel) and lecorguille solution helped. Inside the currently used conda directory (should be in the error log or just look for the most recent folder) inside .snakemake/conda/<hashhere> and edit the file .snakemake/conda/<hashhere>/etc/conda/activate.d/activate-r-base.sh and comment out the line described. This may not work with your setup, but "worked for me".

From
R CMD javareconf > /dev/null 2>&1 || true
to
# R CMD javareconf > /dev/null 2>&1 || true
Best Regards

I am experimenting the same issue... I have not tested your solution, mine was to remove the environment to force it to be reinstalled. I will try yours when I have time. Maybe something for the snakemake devloppers to look at? @johanneskoester ?

johanneskoester commented 5 years ago

So @mingwandroid, it boils down to having something like an environment-wide lock that is acquired before the activate script, ensuring that there are no two activate scripts executed at the same time, right? I would think that this could simply be implemented inside conda activate. Am I getting this right?

mingwandroid commented 5 years ago

Yes. Making the lock as short lived as possible would be ideal

johanneskoester commented 5 years ago

Great. Is this something that somebody on your side wants to do, or would you rather like me to provide a PR? If it should be me, can you give me a pointer where that would be in the code base? Sorry for asking, I don't want to bother you more than necessary, but my time is very limited these days.

mingwandroid commented 5 years ago

Mine is also limited. Ping @msarahan

johanneskoester commented 5 years ago

@msarahan, what do you think?

mbargull commented 5 years ago

I took only a quick look at this. IIUC, the activation script is just to give some users the convenience of having R "automagically" (re-)configure itself according to the currently available (external) Java every time an environment is activated. So, okay, this may be beneficial for some users (i.e., those that change their Java configuration after this package has been installed, but don't care to run javareconf themselves). But frankly, in my opinion, putting this in an activation script is a horrible hack.

My general advice on (de-)activation scripts:

1. avoid creating them whenever you can accomplish your goal in other, saner manners (e.g., try inferring invariants at build or install time instead of re-evaluating things during environment activation), 2. only alter volatile state (e.g., environment variables and such), 3. do proper cleanup in deactivation scripts (e.g., if possible, be non-disruptive: revert only the changes from your activation script -- no more, no less), 4. don't expect your deactivation script to be run in all cases (e.g., closing a shell session will not run them), 5. if you need to save temporary state, try to use somewhat "unique" environment variable names (e.g., specific to the package, possibly shell session etc. => think about how to accomplish point 3. above under the assumptions that other packages' (de-)activation scripts get run, plus "Conda environment stacking" might be a thing..), 6. if environment variables don't cut it for you, try temporary storage, e.g., `mktemp` and friends (can be tricky due to non-POSIX-y stuff, permission issues, etc. => might need fallbacks if `mktemp` fails or something), 7. try to adhere to point 2. (possibly by doing point 1.!), 8. if you *really* might need to change some global/persistent state: 8.1. avoid unnecessary writes (e.g., if the file `state`'s content already is `running`, you needn't `echo 'running' > state`) 8.2. prepare for possible concurrent access (e.g., use locks or, if possible, atomic operations) 9. (there are possible more things I currently didn't think of) ... be fast, be safe (e.g., try writing POSIX-compatible scripts that work with `sh -ue`), ...

Getting back to the the case at hand: IMO, true solutions to the problem are one of the following:

Change R (or whatever is responsible for the R <-> Java interaction) to not depend on persistent state,
Let the user take care of running `javareconf.

I have no idea how R works (and am not really interested in that anyway), hence I can't comment if 1. is possible / feasible. To make 2. work, you can either 2.1. Outright remove the javareconf call in the activation script and tell the users to manage their R-Java stuff themselves, 2.2. Copy the javareconf script and replace all of its writes by simple checks for changed files. If a change is detected, output a message to the user to run javareconf themselves.

Those are disruptive changes, of course. I won't expect 1. to happen. However, 2. is a sane compromise in my book. But since I don't really care much, I won't argue or push for that change.

Adding locks to Conda environments during activation is not an option for me. Having conda lock environments during its own operation (i.e., install/update/remove actions) makes sense. Environment activation on the other hand should not mess with persistent state if ever possible, so locking an environment doesn't make sense for this. If there are some "black sheep" activation scripts like the one of this package, then those scripts should handle concurrent access themselves, not conda.

Imagine if the javareconf script didn't change a file inside the Conda environment itself but in $HOME/.config or the like. If we created two different environments with r-base and concurrently activate them, then locking either of the environments doesn't help at all because the writes happen at some outside location. => conda cannot know if and where activation scripts change global state => conda can't know what to lock => activation scripts should handle concurrent writes themselves.

I'll open a pull request to address "8.1. avoid unnecessary writes" from my recommendation list above. NB: The patch at https://github.com/conda-forge/r-base-feedstock/blob/9934a07a64d4115c9e2bbc50129c4b6eb6bd0bb6/recipe/0017-Foribly-remove-then-forcibly-mv-in-javareconf.in.patch#L20 increases the likely hood of this issue appearing.

mingwandroid commented 5 years ago

IMHO while we have no have Java ecosystem to speak of, allowing different ones per env is important, no necessary. We can do better around our locking here but I don't think we should throw the baby out with the bath water.

These are decisions for R upstream. I didn't write R CMD javareconf or come up with the idea, but I believe per env activation is right. Unless you can carefully articulated why not then we're at an impasse on that point. Please try our R with a range of Javas. I went to extreme lengths to maintain good compatibility for our users because we have nothing much for them ourselves.

I'm not so busy on R these days FWIW but I don't think this is the right track to take.

mingwandroid commented 5 years ago

I disagree with your point that activation scripts shouldn't write files beyond accepting they shouldn't exist when not necessary. Writing env local files is appropriate but we should lock things better.

At the end of the day R provides this facility and I thought it there to take advantage of, that this was most appropriate place (think people experimenting with different Javas with R) and we should be as dynamic and friendly to all the Javas as possible and I believe using it is entirely reasonable (if bugged currently .. be aware we have a few openjdks too, conda-forge's coming from Azul still I think and ADs coming from Jetbrains, and I want to support system and legacy Javas).

I'm just trying to broaden our appeal, I don't use Java at all and R little (ironically usually when trying to fix issues with rJava, yes, largely to do with trying to be broadly compatible (and not surprise or burden our users - which Java should be active? How do I do that?). I don't know how much of R's Java goes through rJava. If all, then perhaps someone could propose moving that functionality into rJava itself, I'd be supportive of that if it's appropriate. That we must people could ignore this (and we can make it work right for other packages that much want to write env local lconfig files during activation).

mbargull commented 5 years ago

[...] allowing different ones per env [...] [...] but I believe per env activation is right [...]

I don't disagree with the "per env" part, just the "activation" one. My stance is just that reconfiguration during every environment activation is excessive and can lead to concurrency issues during writes (i.e., this very issue). I'd rather have those changes made on demand, which means when r-base is installed/updated (which is already the case via the post-link script) or by the user.

I went to extreme lengths to maintain good compatibility for our users because we have nothing much for them ourselves.

Glad to hear, I'll just take your word on this ;).

Please try our R with a range of Javas.

I'll leave this to others since I'm with you with "I don't use Java at all and R little" -- for me it is rather "I don't use R at all and Java little" ;).

I disagree with your point that activation scripts shouldn't write files beyond accepting they shouldn't exist when not necessary.

The thing is just that I believe users likely won't expect environment activations to do much special things, esp. not things that might fail. Plus, authors of activation scripts have to be aware of possible concurrency issues and handle them accordingly.

Writing env local files is appropriate but we should lock things better.

IMO, only in exceptional circumstances for activation scripts. And due to the "exceptional" part, I'd rather not have conda itself deal with this (rather just have the activation script at least do something like lockfile "${CONDA_PREFIX}/path/to/file.lock" ; write-to-file "${CONDA_PREFIX}/path/to/file" ; rm -f "${CONDA_PREFIX}/path/to/file.lock").

Overall my opinion is just

2.2. Copy the javareconf script and replace all of its writes by simple checks for changed files. If a change is detected, output a message to the user to run javareconf themselves.

[...] is a sane compromise in my book. But since I don't really care much, I won't argue or push for that change.

and hence I think for now we should just try to reduce the concurrent write situation here and, if someone is willing to dig into it (i.e., not me), add some proper file locking in javareconf or the activation script.

mingwandroid commented 5 years ago

On gitter people are expecting to be able to operate on the same conda env from multiple processes at the same time and that's what is happening here. I'd rather we fixed it higher level. In no way should an env been mutable by two conda at once. I'd this bug helps us to shake out issues there then that's a win.

But yes, happy for wanting reasonable here too, just want to be flexible and am very concerned about UX and compat.

epruesse commented 5 years ago

Moving the conversation from gitter over here. My points were:

Concurrently entering (activating) the same environment from multiple processes, possibly running on different computers with a shared filesystem, is a pretty common use case. A classic example is parallel execution of jobs on a cluster, e.g. using snakemake as presented above.
An environment changing underneath running processes because it is activated from elsewhere is clearly unexpected behavior. It will break things. Not every job starts R right away, and this kind of thing is guaranteed to break at the most unfortunate time.
Concurrently running conda install or other operations obviously altering an environment OTOH is expected to either break (current behavior), fail or block. It is also reasonably easy to avoid this. It would be awesome to have locking for package cache and env to allow safe concurrent calls to those, but there is no way that two things are adding packages to the same env at the same time.
Locking across operating systems and shared filesystems is finicky. Even highly polished tools such as sqlite3 discourage accessing the same database from multiple hosts. So using locking needs to come with big warning signs.

I would therefore argue that conda activate should be free of side effects.

If things specific to the local host really really really need to be altered, the respective configuration would have to be placed in a temporary directory unique to the activation session. We'd also have to accept that it would not be reliably cleaned away in all cases.

Using locking to protect from things changing underneath running processes (which would lead to undefined results) would require acquiring the lock in activate and releasing it in deactivate. The latter doesn't necessarily happen explicitly though, so that's the first major problem. The other is that the env can then no longer be used in parallel, meaning a cluster job running 50 nodes would need 50 copies of the env. Possible, but a very ugly brute force solution.

In the case at hand, I don't see why host local JVMs need to be supported at all. IMO, the root package requiring the javareconf files should require a JVM installed into the environment and handle the ldpath setup in post-install.sh. That way, it's set up at the time where modifying the env is expected, and the env remains static during use.

In summary, my $0.02 based on experience parallelizing C/C++ and Python apps is that activate should be "const", i.e. idempotent and without side-effects. Everything else will either be slow (=one process per env) or turn out to be a nightmare with constant breakage and patch after patch as more and more corner cases turn up.

mingwandroid commented 5 years ago

If environment mutation is locked which I think it should be and am not sure if that's a bug or a missing feature, then that would include activation (because we've always allowed env mutation, and this problem can happen with anything, env vars for example).

If we fix all that then this is automatically fixed. Though @mbargull's PR will help in the meantime.

msarahan commented 5 years ago

I would really like if activation were assumed to be "const." I agree with Ray, though, that we have the existing expectations to contend with. Perhaps with https://github.com/conda/conda/pull/8727 we can have a "safe" way of activation: if any arbitrary shell scripts are present, it is considered unsafe (mutation may happen), and must be locked. If no unsafe scripts are present, though, conda will not require locking for activation, and perhaps can go faster.

mingwandroid commented 5 years ago

Using locking to protect from things changing underneath running processes (which would lead to undefined results) would require acquiring the lock in activate and releasing it in deactivate

@epruesse, it absolutely would not.

hdraisma commented 4 years ago

I was wondering, would it be possible to give an update re (the resolution of) this issue?

Thank you very much in advance

nehaljwani commented 4 years ago

I hit this problem when hundreds of processes tried to activated a single read only conda environment and spawned hundreds of Java processes, which ended up consuming system resources.

I have a suggestion for this situation which follows up on @mbargull 's second point:

Let the user take care of running `javareconf.

And it is least disruptive to the current setup, because it doesn't change the the status quo.

In the activation script, introduce an env var like CONDA_R_IGNORE_JAVARECONF. If this is set, then the activation script shouldn't run R CMD javareconf > /dev/null 2>&1 || true. That way end users of clusters (where environments don't change that often or have nothing to do with Java) can simply export this env var and activation with be quicker and there will be no mayhem.

freekvh commented 3 years ago

Dear all, I also run into this issue frequently when processing a lot of files in parallel using snakemake with--use-conda (on an NFS share). I see that there is a fix: https://github.com/kpalin/r-base-feedstock/commit/9eda35bdc8ea2c2433cbc6b94c2e978b4d7cd8d4. But since this issue is still open, does that mean it is not in any release yet?

My apologies for my ignorance, I'm not able to determine if that commit is in any release...

By the way, setting my conda envs as read only would be an option for me (actually I'd prefer it, and keep their management restricted to 1 or 2 accounts only).

conda-forge / r-base-feedstock

lib/R/etc/ldpaths: No such file or directory #67