Closed szcf-weiya closed 2 months ago
But the above issue does not occur in Julia 1.7 and even Julia 1.5.
What exactly doesn't occur in Julia v1.7 or 1.5? I don't see how you could not get the same error without preloading.
@giordano That is quite my confusion. Since I ran the same workflow in the GitHub actions with different Julia versions. It turns out that only Julia 1.8 failed. So I thought there should be something different in Julia 1.8.
I asked what exactly doesn't occur, I don't think your answered the question :slightly_smiling_face:
Sorry for the confusion. I mean no error when loading RCurl package in Julia
julia> @rlibrary("RCurl")
I have run into this same issue, and did a bit of digging today (though I'm well out of my depth on this, so apologies in advance for any obvious mistakes).
Here's a quick Dockerfile way to recreate the problem. You can also verify that this problem didn't occur on 1.7 and earlier by changing FROM julia:latest
=> FROM julia:1.7
FROM julia:latest
RUN apt-get -y update && \
apt-get -y install ssh build-essential r-base libcurl4-openssl-dev
# user
RUN useradd --create-home -u 999 --shell /bin/bash newuser
RUN mkdir /home/newuser/app
COPY . /home/newuser/app
WORKDIR /home/newuser/app
RUN chown newuser:newuser -R /home/newuser/app
USER newuser
RUN julia -e "using Pkg; Pkg.activate(\".\"); Pkg.add(\"RCall\"); Pkg.add(\"HTTP\");"
# Install R packages
RUN mkdir -p /home/newuser/R/site-library
ENV R_LIBS_USER "/home/newuser/R/site-library"
RUN Rscript -e 'install.packages(c("RCurl"), lib = "/home/newuser/R/site-library")'
RUN Rscript -e 'require("RCurl")';
ENTRYPOINT ["/bin/bash"]
Can then recreate the import failure by:
julia> using RCall, HTTP
julia> @rimport RCurl
ERROR: REvalError: Error in dyn.load(file, DLLpath = DLLpath, ...) :
unable to load shared object '/home/newuser/R/site-library/RCurl/libs/RCurl.so':
/usr/local/julia/bin/../lib/julia/libcurl.so: version `CURL_OPENSSL_4' not found (required by /home/newuser/R/site-library/RCurl/libs/RCurl.so)
Running R by itself (not from within Julia) has no issues with library(RCurl)
.
I think the problem is RCurl is trying to link to the version of libcurl Julia has (ie /usr/local/julia/bin/../lib/julia/libcurl.so
). But that doesn't have a libcurl.so.4
:
julia> run(`ldd /usr/local/julia/bin/../lib/julia/libcurl.so`)
linux-vdso.so.1 (0x00007fffa45cb000)
libnghttp2.so.14 => /usr/local/julia/bin/../lib/julia/libnghttp2.so.14 (0x00007f4883400000)
libssh2.so.1 => /usr/local/julia/bin/../lib/julia/libssh2.so.1 (0x00007f4883000000)
libmbedtls.so.14 => /usr/local/julia/bin/../lib/julia/libmbedtls.so.14 (0x00007f4882c00000)
libmbedx509.so.1 => /usr/local/julia/bin/../lib/julia/libmbedx509.so.1 (0x00007f4882800000)
libmbedcrypto.so.7 => /usr/local/julia/bin/../lib/julia/libmbedcrypto.so.7 (0x00007f4882400000)
libz.so.1 => /usr/local/julia/bin/../lib/julia/libz.so.1 (0x00007f4882000000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f4883b4c000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4883b47000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f488221f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4883b5b000)
Process(`ldd /usr/local/julia/bin/../lib/julia/libcurl.so`, ProcessExited(0))
Explicitly setting LD_LIBRARY_PATH="/lib/x86_64-linux-gnu"
seems to encourage R to use the system installed version of libcurl, instead of Julia's. And that seems to solve the problem. I am guessing (though unsure) that this is what was happening on Julia 1.7 and earlier.
What actually changed is that Julia 1.8.4 switched to using RUNPATH instead of RPATH (which was used in Julia 1.8.3 and earlier) for specifying local library paths:
├ objdump -x /central/software/julia/1.8.4/bin/julia |grep 'R.*PATH'
RUNPATH $ORIGIN/../lib:$ORIGIN/../lib/julia
├ objdump -x /central/software/julia/1.8.3/bin/julia |grep 'R.*PATH'
RPATH $ORIGIN/../lib:$ORIGIN/../lib/julia
(since libcurl.so
is in <juliadir>/lib/julia
)
From https://amir.rachum.com/shared-libraries/:
The only difference between
rpath
andrunpath
is the order they are searched in. Specifically, their relation toLD_LIBRARY_PATH
-rpath
is searched in beforeLD_LIBRARY_PATH
whilerunpath
is searched in after. The meaning of this is thatrpath
cannot be changed dynamically with environment variables whilerunpath
can.
@simonbyrne Thanks for pointing out the search order difference between rpath
and runpath
. I also caught the difference, but failed to get the logic of the error. (And the same error also occurs on Julia1.8.1 after retesting)
In Julia 1.7.0, with rpath
, we can properly call RCurl
, which relies on the system's libcurl.so
In Julia 1.8.4, with runpath
, calling RCurl
would call Julia's libcurl.so
BTW, I just retest it in Julia 1.8.1, calling RCurl
throws the same error as the one in Julia1.8.4
julia> using RCall
julia> ENV["R_HOME"]
"/usr/lib/R"
julia> @rlibrary("RCurl")
ERROR: REvalError: Error in dyn.load(file, DLLpath = DLLpath, ...) :
in all cases, RCall.jl is based on the same system R /usr/lib/R
.
Hmm, so maybe the problem is caused by something else.
I'm closing this as stale. If it's still a problem on Julia 1.10, then please open another issue.
I am using the latest v0.13.14
RCall.jl
with R 4.2.1 and Julia 1.8.4 on Ubuntu 20.04.It works well but I found a problem related to libcurl.so.
I installed
RCurl
package in R, it works well, and theldd
status is as follows, which depends on/lib/x86_64-linux-gnu/libcurl.so.4
However, when I call it in Julia 1.8, it throws
If I understand correctly, it wants to use
/opt/hostedtoolcache/julia/1.8.4/x64/bin/../lib/julia/libcurl.so
instead of/lib/x86_64-linux-gnu/libcurl.so.4
.I have checked
ldd
inside the Julia session and the R-in-Julia session, both still show/lib/x86_64-linux-gnu/libcurl.so.4
I found a workaround, set
LD_PRELOAD=/lib/x86_64-linux-gnu/libcurl.so.4
before launching Julia. But a drawback later I found is that it would bring new problems when I useHTTP
package, which also depends on libcurl.so, so if I pre-load the system version, it would break down the HTTP package which relied on Julia's libcurl.soBut the above issue does not occur in Julia 1.7 and even Julia 1.5.
After checking the NEWs of Julia 1.8, https://docs.julialang.org/en/v1/NEWS/ I guess it might be related to
but I am not sure. I tried to understand from the
rpath
viewpoint, but failed to get the logic why loadingRCurl
R package would be conflicted with Julia's libcurl.so.Or maybe it is unrelated to the
rpath
part, but if so, why older Julia 1.7 works? Which change in Julia 1.8 caused the difference?Hope anyone can help explain it. Many thanks!