running Rstudio container with podman

[FelixErnst]

Hi,

I played around with buildah and podman and tried to get a rocker container to work with podman.

However, when I ran the container and attached the following error kept on repeating, which suggest some start up problem of rstudio service.

3 Jan 2020 21:33:46 [rserver] ERROR system error 1 (Operation not permitted); OCCURRED AT: rstudio::core::Error rstudio::core::system::setResourceLimit(rstudio::core::system::ResourceLimit, RLimitType, RLimitType) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/core/system/PosixSystem.cpp:1496; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/server/ServerMain.cpp:478
rserver[9447]: ERROR system error 1 (Operation not permitted); OCCURRED AT: rstudio::core::Error rstudio::core::system::setResourceLimit(rstudio::core::system::ResourceLimit, RLimitType, RLimitType) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/core/system/PosixSystem.cpp:1496; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/server/ServerMain.cpp:478
rsession: no process found

Any suggestion, what might cause this and how to solve the problem?

Thanks for any advice

Container was started like this

podman run -dit --env PASSWORD=bioc -p 8788:8787 --name rocker docker.io/rocker/rstudio

PS: I also tried to build the container with buildah following the commands in the Dockerfile. Upon installing the deb it stop with basically the same error message using the buildah run $rstudio bash -c "dpkg -i rstudio-server-*-amd64.deb" command.

S6_VERSION="v1.21.7.0"
PANDOC_TEMPLATES_VERSION="2.9"
RSTUDIO_URL="https://www.rstudio.org/download/latest/stable/server/bionic/rstudio-server-latest-amd64.deb"

rstudio=$(buildah from docker.io/rocker/r-ver:3.6.2) 

PATH_RSTUDIO=$(buildah run $rstudio printenv PATH)
buildah config --env S6_VERSION=$S6_VERSION --env S6_BEHAVIOUR_IF_STAGE2_FAILS=2 \
    --env PATH=/usr/lib/rstudio-server/bin:$PATH_RSTUDIO --env PANDOC_TEMPLATES_VERSION=$PANDOC_TEMPLATES_VERSION $rstudio

buildah config -p 8787 $rstudio

buildah run $rstudio apt-get update && \
    buildah run $rstudio apt-get install -y --no-install-recommends \
        file \
        git \
        libapparmor1 \
        libclang-dev \
        libcurl4-openssl-dev \
        libedit2 \
        libssl-dev \
        lsb-release \
        multiarch-support \
        psmisc \
        procps \
        python-setuptools \
        sudo \
        wget \
        gdebi && \
    buildah config --workingdir /tmp $rstudio && \
    buildah run $rstudio wget -q $RSTUDIO_URL && \
    buildah run $rstudio bash -c "dpkg -i rstudio-server-*-amd64.deb"

> Selecting previously unselected package rstudio-server.
> (Reading database ... 34584 files and directories currently installed.)
> Preparing to unpack rstudio-server-latest-amd64.deb ...
> Unpacking rstudio-server (1.2.5033) ...
> Setting up rstudio-server (1.2.5033) ...
> groupadd: group 'rstudio-server' already exists
> rserver[52]: ERROR system error 1 (Operation not permitted); OCCURRED AT: rstudio::core::Error rstudio::core::system::setResourceLimit(rstudio::core::system::ResourceLimit, RLimitType, RLimitType) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/core/system/PosixSystem.cpp:1496; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/server/ServerMain.cpp:478

[cboettig]

I think @nuest has some experience running rocker images with podman? I haven't tried it myself. This may be related to https://github.com/rocker-org/rocker/issues/348.

We'll be trying to better integrate & improve community documentation in the coming months (cc @noamross). It might help if you could share some of your motivations for using podman (e.g. instead of singularity, or the docker daemon with --user if you're looking for a rootless mode) since it will help us better document alternatives as well as solutions.

[FelixErnst]

Thanks for the reply and the link to the other repo.

I would call it personal interest, since the buildah, podman and skopeo tools add quite easy to understand flavor to the container universe. The seperate processes for building an image and running a container are bit easier to grasp in my opinion and the manual are shorter and not as convoluted with different api versions as it is the case with docker.

In addition I like the simplicity of just building your image with a bash script, which allows for quite rich opportunities of customisation from a single source, which can be achieved with docker, but just with a bunch of docker files (with my limited experience at least and without kubernetes).

How I ended up here: I was able to transfer the build process from the Dockerfile of r-ver into a bash script running the build process with buildah, which was resulted in a bit smaller image than the one from docker.io. I wanted to build up on it just for fun with the ultimate goal of adding the Bioconductor flavor ontop of rocker

edit: And in addition podman uses the same syntax as docker for running a container. So for running containers the is not a lot relearning involved, I guess.

[FelixErnst]

So it is about rocker needing root on the host. Running rocker with podman as root does work, which begs the question: Why does it need root on the host?

edit: since the origin of the error occurs somewhere in some C++ code, it has to be a buildin feature of rstudio server. Does this warrant some escalation?

[FelixErnst]

So which type of resource limit is set in the function setResourceLimit? Does anyone know who to tag to this issue or who to ask?

Thanks for any directions.

[eddelbuettel]

From memory you can do it from bash but some limit settings may require sudo.

[FelixErnst]

Sorry, I was not very clear want I meant and sorry again for spreading it over two issues.

I tried setting resource limits with podman in rootless mode up to 12g memory and 8 cpus. Since this is the maximum for the VBox I am using and it works with sudo podman I assume it is about how rstudio server tries to set a resource limit using setResourceLimit which is reported as the culprit in the original error message.

Maybe setResourceLimit also tries to set a resource limit different from memory or cpus, but that is hard to guess since there is no mention of the RLimitType in the error message. I suppose the source code for rstudio server is not available directly, I cannot check, what might be used in ServerMain.cpp:478

So thats why ended with asking: So which type of resource limit is set in the function setResourceLimit? Does anyone know who to tag to this issue or who to ask?

@jspiewak @colearendt @jcheng5 @meatballhat @rich-iannone: any idea who to talk to about this? Any interest in investigating?

Thanks for any help.

[jspiewak]

The open source version of RStudio is available here.

For the version of RStudio you appear to be running, that would be ServerMain.cpp:478:

      // increase the number of open files allowed (need more files
      // so we can supports lots of concurrent connectins)
      if (core::system::realUserIsRoot())
      {
         Error error = setResourceLimit(core::system::FilesLimit, 4096);
         if (error)
            return core::system::exitFailure(error, ERROR_LOCATION);
      }

[FelixErnst]

Thanks you.

I assumed that RStudio != RStudio server, but thank you for pointing out the repo.

[FelixErnst]

So this work using the following options. Thanks to all for pointing me in the right direction.

works

podman run -dit --ulimit="nofile=4096" --env PASSWORD=bioc -p 8788:8787 --name rocker docker.io/rocker/rstudio

does not work

podman run -dit --ulimit="nofile:4095" --env PASSWORD=bioc -p 8788:8787 --name rocker docker.io/rocker/rstudio

Using buildah

rstudio=$(buildah from --ulimit="nofile=4096" $IMAGE)

[cboettig]

@FelixErnst very cool, thanks much for sharing the results of your investigation here!

[nuest]

Thanks for the ping @cboettig - I made some experiments a while back, all documented in https://github.com/nuest/rodman

I got plain R working, but ran into permission problems with RStudio, so great to see @FelixErnst could solve those!

I added a link to this issue in my repo's README file and am looking forward to the community documentation coming up and will be happy to contribute.

[sahilseth]

update: the issue I have, better relates to rocker-org/rocker-versioned2#373, than this issue.

I am getting a similar error with singularity. Interestingly the same cmd worked a few weeks back, but gives an error with the latest pull:

singularity exec ~/singularity-images/rstudio_geospatial_4.0.0.simg rserver --www-port 57584 --auth-none=0 --auth-pam-helper-path=pam-helper

09 Jun 2020 00:21:30 [rserver] ERROR system error 11 (Resource temporarily unavailable) [description: Could not acquire revocation list file lock]; OCCURRED AT rstudio::core::Error rstudio::server::auth::handler::initialize() src/cpp/server/auth/ServerAuthHandler.cpp:570; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:674

I tried to see the ulimit like @FelixErnst suggested, it seems to be set to unlimited:

singularity shell $rstudio_simg
Singularity> ulimit
unlimited

Don't think it is related, but I saw these warning when creating the image:

2020/06/08 19:39:13  warn rootless{opt/texlive/texmf-var/fonts/map/pdftex/updmap/pdftex.map} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:21  warn rootless{usr/lib/x86_64-linux-gnu/gstreamer1.0/gstreamer-1.0/gst-ptp-helper} ignoring (usually) harmless EPERM on setxattr "security.capability"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/dvilualatex} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/dviluatex} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/dvipdfm} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/dvipdfmx} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/ebb} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/etex} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/extractbb} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/fmtutil} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/fmtutil-sys} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/fmtutil-user} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/inimf} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/initex} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/latex} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/lualatex} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/luaotfload-tool} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/man} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/mktexfmt} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/mktexlsr} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/mktexmf} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/mktexpk} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/mktextfm} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/pdfcrop} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/pdfetex} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/pdflatex} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/rpdfcrop} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/rungs} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/simpdftex} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/texhash} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/texlua} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/texluac} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/tlmgr} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/updmap} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/updmap-sys} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/updmap-user} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:26  warn rootless{usr/local/texlive/bin/x86_64-linux/xelatex} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2020/06/08 19:39:48  info unpack layer: sha256:c6b684197d71c0800c87c1cdea1945cae1abf1d56e442e3a8587f8bec20c7e71

[FelixErnst]

singularity shell $rstudio_simg
Singularity> ulimit
unlimited

This doesn't mean a thing, if singularity from the outside uses a limit.

The solution was to direct podman what limits to use and not the container system itself. See my examples above.

So it might well be, that for singularity a limit was added. Best place to start is to report to them and not the image maintainer and look for the settings for singularity managing the limits