Docker image build problem: couldn't find packages from bioconductor

[YonghuiDong]

My dockerfile was build with

golem::add_dockerfile()

Here is the docker file:

FROM rocker/verse:4.2.0
RUN apt-get update && apt-get install -y  cmake git-core libcurl4-openssl-dev libgit2-dev libglpk-dev libgmp-dev libicu-dev libpng-dev libssl-dev libxml2-dev make pandoc zlib1g-dev && rm -rf /var/lib/apt/lists/*
RUN mkdir -p /usr/local/lib/R/etc/ /usr/lib/R/etc/
RUN echo "options(repos = c(CRAN = 'https://cran.rstudio.com/'), download.file.method = 'libcurl', Ncpus = 4)" | tee /usr/local/lib/R/etc/Rprofile.site | tee /usr/lib/R/etc/Rprofile.site
RUN R -e 'install.packages("remotes")'
RUN Rscript -e 'remotes::install_version("tidyr",upgrade="never", version = "1.2.1")'
RUN Rscript -e 'remotes::install_version("RColorBrewer",upgrade="never", version = "1.1-3")'
RUN Rscript -e 'remotes::install_version("dplyr",upgrade="never", version = "1.0.10")'
RUN Rscript -e 'remotes::install_version("shiny",upgrade="never", version = "1.7.4")'
RUN Rscript -e 'remotes::install_version("rmarkdown",upgrade="never", version = "2.19")'
RUN Rscript -e 'remotes::install_version("ggplot2",upgrade="never", version = "3.4.0")'
RUN Rscript -e 'remotes::install_version("igraph",upgrade="never", version = "1.3.5")'
RUN Rscript -e 'remotes::install_version("waiter",upgrade="never", version = "0.2.5")'
RUN Rscript -e 'remotes::install_version("shinydashboard",upgrade="never", version = "0.7.2")'
RUN Rscript -e 'remotes::install_version("config",upgrade="never", version = "0.3.1")'
RUN Rscript -e 'remotes::install_version("VIM",upgrade="never", version = "6.2.2")'
RUN Rscript -e 'remotes::install_version("shinydashboardPlus",upgrade="never", version = "2.0.3")'
RUN Rscript -e 'remotes::install_version("ropls",upgrade="never", version = "1.30.0")'
RUN Rscript -e 'remotes::install_version("readxl",upgrade="never", version = "1.4.1")'
RUN Rscript -e 'remotes::install_version("readr",upgrade="never", version = "2.1.3")'
RUN Rscript -e 'remotes::install_version("qgraph",upgrade="never", version = "1.9.3")'
RUN Rscript -e 'remotes::install_version("plotly",upgrade="never", version = "4.10.1")'
RUN Rscript -e 'remotes::install_version("golem",upgrade="never", version = "0.3.5")'
RUN Rscript -e 'remotes::install_version("ggraph",upgrade="never", version = "2.1.0")'
RUN Rscript -e 'remotes::install_version("DT",upgrade="never", version = "0.26")'
RUN Rscript -e 'remotes::install_version("ComplexHeatmap",upgrade="never", version = "2.12.1")'
RUN mkdir /build_zone
ADD . /build_zone
WORKDIR /build_zone
RUN R -e 'remotes::install_local(upgrade="never")'
RUN rm -rf /build_zone
EXPOSE 80
CMD R -e "options('shiny.port'=80,shiny.host='0.0.0.0');MetReport::run_app()"

It seems that remote::remotes::install_version() cannot install R packages from bioconductor.

The error message:

 > [18/31] RUN Rscript -e 'remotes::install_version("ropls",upgrade="never", version = "1.30.0")':
ThinkR-open/golem#21 9.175 Error in download_version_url(package, version, repos, type) : 
ThinkR-open/golem#21 9.175   couldn't find package 'ropls'
ThinkR-open/golem#21 9.175 Calls: <Anonymous> -> download_version_url
ThinkR-open/golem#21 9.176 Execution halted

My shinyapp Description file was generated by golem.

How should I solve this problem?

Thanks a lot.

Dong

[ColinFay]

(transfering to Dockerfiler, as it handles the engineering of apps)

[VincentGuyader]

can you tyr with golem::add_dockerfile_with_renv() ?

[YonghuiDong]

Dear @VincentGuyader and @ColinFay ,

Thanks a lot for your prompt and kind help. I have tried with golem::add_dockerfile_with_renv(output_dir = "deploy"), and I encountered the following error when I build the docker image:

invalid argument "MetReport_base" for "-t, --tag" flag: invalid reference format: repository name must be lowercase

Does that mean that I have to change my shinyapp name in lowercase?

Below is the generated dockerfile

FROM MetReport_base
COPY renv.lock.prod renv.lock
RUN R -e 'renv::restore()'
COPY MetReport_*.tar.gz /app.tar.gz
RUN R -e 'remotes::install_local("/app.tar.gz",upgrade="never")'
RUN rm /app.tar.gz
EXPOSE 80
CMD R -e "options('shiny.port'=80,shiny.host='0.0.0.0');MetReport::run_app()"

Best regards, Dong

[VincentGuyader]

what is your golem version ? can you try to install the last golem version i have corrected this issue : https://github.com/ThinkR-open/golem/issues/944 remotes::install_github("ThinkR-open/golem@dev")

[YonghuiDong]

My golem version is 0.3.5.

After updating to 0.4.0 , I still get the same error message when I using the follow script to build the image.

docker build -f Dockerfile_base --progress=plain -t MetReport_base .
docker build -f Dockerfile --progress=plain -t MetReport:latest .
docker run -p 80:80 MetReport:latest
# then go to 127.0.0.1:80

Error message:

invalid argument "MetReport_base" for "-t, --tag" flag: invalid reference format: repository name must be lowercase

[VincentGuyader]

we made a mistake during our github merge request, I will check it right now, but please can you try with : remotes::install_github("ThinkR-open/golem@89b40cb568ed2d8edd75909cbbf7dc693f25b82f")

edit : after checking more deeper, are you sure to have restarted your R after golem installation ?

i made a test with remotes::install_github("ThinkR-open/golem@dev") on a golem called "MAJUSCULE" and I got a fully functionnal Dockerfile

[YonghuiDong]

Thanks a lot @VincentGuyader , the image was built now using docker build -f Dockerfile_base --progress=plain -t MetReport_base .

1. But it took 3.5 h to create the image, and the image size is 5.3G. I have about 25 direct dependency packages, but I did not expect the image size to be that big.

2. I cannot continue with the next docker script docker build -f Dockerfile --progress=plain -t MetReport:latest ., and I got the following error:

failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed.

3. I will try to build it manually, and update you later on.

Best regards, Dong

[YonghuiDong]

Dear @VincentGuyader,

I have manually edited the dockerfile created using golem::add_dockerfile(), in which I used BiocManager to install R packages from Bioconductor.

FROM rocker/verse:4.2.0
RUN apt-get update && apt-get install -y  cmake git-core libcurl4-openssl-dev libgit2-dev libglpk-dev libgmp-dev libicu-dev libpng-dev libssl-dev libxml2-dev make pandoc zlib1g-dev && rm -rf /var/lib/apt/lists/*
RUN mkdir -p /usr/local/lib/R/etc/ /usr/lib/R/etc/
RUN echo "options(repos = c(CRAN = 'https://cran.rstudio.com/'), download.file.method = 'libcurl', Ncpus = 4)" | tee /usr/local/lib/R/etc/Rprofile.site | tee /usr/lib/R/etc/Rprofile.site

# Install bioconductor packages
FROM bioconductor/bioconductor_docker:RELEASE_3_16
RUN apt-get update
    RUN R -e 'BiocManager::install(ask = F)' && R -e 'BiocManager::install(c("ropls", "ComplexHeatmap", ask = F))'

# Install Cran and other packages
RUN R -e 'install.packages("remotes")'
RUN Rscript -e 'remotes::install_version("tidyr",upgrade="never", version = "1.2.1")'
RUN Rscript -e 'remotes::install_version("RColorBrewer",upgrade="never", version = "1.1-3")'
RUN Rscript -e 'remotes::install_version("dplyr",upgrade="never", version = "1.0.10")'
RUN Rscript -e 'remotes::install_version("shiny",upgrade="never", version = "1.7.4")'
RUN Rscript -e 'remotes::install_version("rmarkdown",upgrade="never", version = "2.19")'
RUN Rscript -e 'remotes::install_version("ggplot2",upgrade="never", version = "3.4.0")'
RUN Rscript -e 'remotes::install_version("igraph",upgrade="never", version = "1.3.5")'
RUN Rscript -e 'remotes::install_version("waiter",upgrade="never", version = "0.2.5")'
RUN Rscript -e 'remotes::install_version("shinydashboard",upgrade="never", version = "0.7.2")'
RUN Rscript -e 'remotes::install_version("config",upgrade="never", version = "0.3.1")'
RUN Rscript -e 'remotes::install_version("remotes",upgrade="never", version = "2.4.2")'
RUN Rscript -e 'remotes::install_version("VIM",upgrade="never", version = "6.2.2")'
RUN Rscript -e 'remotes::install_version("shinydashboardPlus",upgrade="never", version = "2.0.3")'
#RUN Rscript -e 'remotes::install_version("ropls",upgrade="never", version = "1.30.0")'
RUN Rscript -e 'remotes::install_version("readxl",upgrade="never", version = "1.4.1")'
RUN Rscript -e 'remotes::install_version("readr",upgrade="never", version = "2.1.3")'
RUN Rscript -e 'remotes::install_version("qgraph",upgrade="never", version = "1.9.3")'
RUN Rscript -e 'remotes::install_version("plotly",upgrade="never", version = "4.10.1")'
RUN Rscript -e 'remotes::install_version("ggraph",upgrade="never", version = "2.1.0")'
RUN Rscript -e 'remotes::install_version("DT",upgrade="never", version = "0.26")'
RUN Rscript -e 'remotes::install_version("markdown",upgrade="never", version = "1.4")'
#RUN Rscript -e 'remotes::install_version("ComplexHeatmap",upgrade="never", version = "2.12.1")'
RUN Rscript -e 'remotes::install_github("ThinkR-open/golem@89b40cb568ed2d8edd75909cbbf7dc693f25b82f")'
RUN mkdir /build_zone
ADD . /build_zone
WORKDIR /build_zone
RUN R -e 'remotes::install_local(upgrade="never")'
RUN rm -rf /build_zone
EXPOSE 80
#CMD R -e "options('shiny.port'=80,shiny.host='0.0.0.0');MetReport::run_app()"
CMD R -e "options('shiny.port'=80,shiny.host='0.0.0.0')"

This time it took about 50 min to build the image, and the image size is about 4.7G.

But when I tried to run the image docker run -p 80:80 metaboreport. No error was popped up, but the container stopped immediately.

I am using M1 Mac, so I used docker buildx build --platform linux/amd64 -t metaboreport . to build docker image.

Do you happen to know the possible reason?

Best regards, Dong

[YonghuiDong]

Dear @VincentGuyader,

I have finally rebuilt the docker image using golem::add_dockerfile_with_renv(), and it worked eventually.

I am using M1 Mac, so it is much slower in building a docker image, and I need to add --platform=linux/amd64 flag when building the image.

Thanks a lot again for your kind help.

Dong

[remlapmot]

RE: the point about Apple Silicon Macs running slowly under the old emulation. About 1 week ago Docker released a new beta feature for Apple Silicon Macs - it's called Rosetta for Linux - you need to update your Docker to version 4.16.0 or later to get it.

It's off by default. Once enabled in Docker settings (select the box in the screenshot below and click "Apply & restart") I have experienced amd64 containers build in literally one third the time compared to the old emulation.

[YonghuiDong]

Hi Tom @remlapmot,

Thanks a lot for the tip, it is very helpful.

Dong