r-base containers :4.1.2, :4.1.3, and latest break with ERROR: R_HOME ('/usr/lib/R') not found

[bockthom]

Hello, I use your official r-base docker containers to run R scripts within a CI pipeline. Basically, I test the scripts on different R-versions using different r-base containers. (I have to use r-base instead of r-vers as I need the most recent package versions and not the one at the last point an R version was most recent.) This worked for quite a long time, but it does not work any more since r-base:4.1.2 (so r-base:4.1.3 and r-base:latest are also affected).

In these containers, calling Rscript script.R ends up in the following error:

Rscript script.R
ERROR: R_HOME ('/usr/lib/R') not found

After lots of hours of debugging, I found out what causes the problem:

As of glibc version 2.33, glibc uses "faccessat2" which was forbidden by docker: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1005906 https://bugzilla.redhat.com/show_bug.cgi?id=1900021 According to here, this bug "has been fixed in docker.io 20.10, which is available in bullseye as well." on 18 Feb 2022.

As your r-base containers starting from version 4.1.2 use glibc version 2.33, this bug also affects your containers. Would it be possible to re-build all of the affected r-base containers such that they contain the above listed bugfix? I know that you usually don't update the containers for already released versions, but I would highly appreciate if you could fix that as otherwise I have trouble with these R-versions (even the current one) in my CI pipeline, and also many other users potentially have.

---

I already searched for workarounds, but none of the ones I found did work for me: (1) https://github.com/clearlinux/distribution/issues/2361#issuecomment-1030946360 Here they say that it would work to remove some affected lines from /usr/bin/R inside the container. I tested that and it worked for calling R, but not for Rscript as Rscript is a binary file, so I cannot workaround the Rscript call. However, in my CI pipeline, I need to call Rscript. (2) Another workaround they propose is to run the container without seccomp: https://github.com/clearlinux/distribution/issues/2361#issuecomment-1032065550 This might work, but not in my case, as the CI pipline prevents me from running the container in a privileged mode. So, this workaround is also not an option.

So, I also searched for other workarounds and tried lots of different things, but all of them resulted in errors. So, to me, there are only two ways how this problem could be fixed: Either install a lower glibc version (< 2.33) inside the container --- this is something you would have to do as it was impossible for me to downgrade the installed glibc version inside the container --- or to build on a fixed docker installation as indicated in the last post of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1005906.

Could you please have a look at this problem and provide fixed versions for r-base:4.1.2, r-base:4.1.3 and r:latest? Otherwise it is impossible to use these containers in a CI pipeline for Rscripts. Thanks in advance!

[eitsupi]

I could not see that error in my environment (latest version of Docker Desktop, 20.10.13, build a224086), so perhaps this is a bug that will be fixed by an update of the Docker engine, as you mentioned in your post? I was wondering if you have a particular reason for not being able to update the Docker engine on your Docker host.

I have to use r-base instead of r-vers as I need the most recent package versions and not the one at the last point an R version was most recent.

Note that this can simply be resolved by switching the factory-set repository.

https://www.rocker-project.org/use/extending/

FROM rocker/r-ver:4.1.2
RUN echo 'options(repos = c(CRAN = "https://cloud.r-project.org"))' >>"${R_HOME}/etc/Rprofile.site"

[eddelbuettel]

I am with @eitsupi here: it also does not fail for me, and I create quite a few derived containers off r-base and I have no issues either:

edd@rob:~$ docker run --rm -ti r-base bash
root@c1cfefc623d3:/# Rscript --version
R scripting front-end version 4.1.3 (2022-03-10)
root@c1cfefc623d3:/# Rscript -e 'print(Sys.getenv("R_HOME"))'
[1] "/usr/lib/R"
root@c1cfefc623d3:/# 

I don't see a course of action here, nor do I see an issue with r-base package set (for which I happen to the mainainer). The standard R invocation used before we enve to R CMD also works:

root@c1cfefc623d3:/# R RHOME
/usr/lib/R
root@c1cfefc623d3:/# 

So so far I see neither I reason to change things going forward and neither to go and retro-fit (which we are unlikely to do for reproducibility reasons). You would have to convince us more strongly that this is generic shortcoming in the container as opposed to a local issue at your end. Also note that the same (base) container (and lots of derived containers) are happily used in many CI setups by different users.

[bockthom]

Thanks for your fast replies!

I could not see that error in my environment (latest version of Docker Desktop, 20.10.13, build a224086), so perhaps this is a bug that will be fixed by an update of the Docker engine, as you mentioned in your post?

I am not sure about that. To me it sounds that this bug affects not really the docker engine but the resulting container at the point of creating a container. So, it sounds like that the r-base container needs to be recreated for the bug to disappear. But, to be honest, I am not into the details of docker, I just stumbled over the bug reports while searching for causes and solutions for the problem.

I was wondering if you have a particular reason for not being able to update the Docker engine on your Docker host.

I don't have a docker host and a docker engine. I just have a drone CI pipeline that runs in a cloud. So, I just specify which docker container and which architecture to use and which tasks and tests should be performed inside the given docker container within the CI pipeline. So I can only debug what's going on inside the docker r-base container (and also not interactively), but I cannot configure how the container is started etc. So I think this problem might be rather related to how the container was created than which docker engine finally runs the container. But again, this was just a guess and I actually don't know.

I have to use r-base instead of r-vers as I need the most recent package versions and not the one at the last point an R version was most recent.

Note that this can simply be resolved by switching the factory-set repository.

https://www.rocker-project.org/use/extending/

FROM rocker/r-ver:4.1.2
RUN echo 'options(repos = c(CRAN = "https://cloud.r-project.org"))' >>"${R_HOME}/etc/Rprofile.site"

Ah this is interesting, I wasn't aware of that. I tried this and it seems to work. So, this does not solve the actual problem, but it looks like a good workaround for now. Thanks for the suggestion.

[eitsupi]

I don't have a docker host and a docker engine. I just have a drone CI pipeline that runs in a cloud. So, I just specify which docker container and which architecture to use and which tasks and tests should be performed inside the given docker container within the CI pipeline. So I can only debug what's going on inside the docker r-base container (and also not interactively), but I cannot configure how the container is started etc. So I think this problem might be rather related to how the container was created than which docker engine finally runs the container. But again, this was just a guess and I actually don't know.

Docker 20.10.0 was released over a year ago. https://docs.docker.com/engine/release-notes/#20100 Therefore, it is unlikely that an older version was used to build r-base:4.1.2.

Is it possible that Drone CI is configured with Docker outside of Docker and the version of that Docker engine is out of date? https://docs.drone.io/runner/docker/installation/linux/

[eitsupi]

https://github.com/clearlinux/distribution/issues/2361#issuecomment-1031967590 As this comment points out, this is most likely due to an older version of the Docker engine running the container.

I recommend that you follow the Docker documentation and install Docker CE (as distinguished from docker.io, which can be installed from the default apt repository). https://docs.docker.com/engine/install/debian/

[mpettis]

I am getting this error as well, though I can see that /usr/lib/R does exist, contrary to the message:

Though after reading, it might be from running a much older version of docker (Docker version 19.03.2, build 6a30dfc). I am totally on a headless machine, recommendations for upgrades that would run this are welcome (I only use this currently to run R stuff for the most part).

[eddelbuettel]

I don't see that:

edd@rob:~$ docker run --rm -ti r-base:4.1.3 Rscript -e 'Sys.getenv()' | grep R_HOME
R_HOME                  /usr/lib/R
edd@rob:~$ docker run --rm -ti r-base:4.1.2 Rscript -e 'Sys.getenv()' | grep R_HOME
R_HOME                  /usr/lib/R
edd@rob:~$ docker run --rm -ti rocker/r-base:latest Rscript -e 'Sys.getenv()' | grep R_HOME
R_HOME                  /usr/lib/R
edd@rob:~$ 

[eddelbuettel]

Can you share briefly what your host OS and version are? I am on Ubuntu 22.04.

[mpettis]

Yes, and thanks. Admittedly, it is very old, and I am trying to get things migrated to an updated OS and docker versions. So I could see this all being closed with, "the OS and docker version are too old" and I have to upgrade...

[mpettis@ip-10-0-200-174 ~]$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:        16.04
Codename:       xenial
[mpettis@ip-10-0-200-174 ~]$ docker --version
Docker version 19.03.2, build 6a30dfc

[eddelbuettel]

Ok, so let's agree that it is a non-issue then under current setups. (Unless proven otherwise, of course so feel free to reopen when you replicate under current settings.)

But as volunteers our resources and time are somewhat limited and do not stretch out infinitely to any release ever made.