Cannot install packages from RStudio container when behind private firewall

AndreaPi commented 5 years ago

On a remote server:

NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

I try to run

docker run -d -p 8787:8787 -e PASSWORD=<mypassword> -e http_proxy=<myproxy> -e https_proxy=<myproxy> -e no_proxy=localhost --name rstudio rocker/tidyverse

It starts nicely, and I can access the RStudio GUI in the browser from my client machine. However, any time I try to install a package, it fails:

> install.packages("janitor")
Warning in install.packages :
  unable to access index for repository https://cran.rstudio.com/src/contrib:
  cannot open URL 'https://cran.rstudio.com/src/contrib/PACKAGES'
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
Warning in install.packages :
  unable to access index for repository https://cran.rstudio.com/src/contrib:
  cannot open URL 'https://cran.rstudio.com/src/contrib/PACKAGES'
Warning in install.packages :
  package ‘janitor’ is not available (for R version 3.5.1)

Following https://support.rstudio.com/hc/en-us/articles/200488488-Configuring-R-to-Use-an-HTTP-or-HTTPS-Proxy, I even tried to edit the .Renviron file with

file.edit('~/.Renviron')

which became

http_proxy=<myproxy>
https_proxy=<myproxy>
no_proxy=localhost

I then restarted R from the Session. No good: I got the same error. Why can't I install any packages inside the RStudio container?

eddelbuettel commented 5 years ago

Can you backtrace what/where it breaks?

Recall that all our Docker containers start from base images and then install packages pretty much the same way. It looks like you have a networking issue here, and it may well be cause by you and your proxy.

Witness:

rob:~$ docker run --rm -ti r-base bash
root@487d901bf794:/# install.r zoo
trying URL 'https://cloud.r-project.org/src/contrib/zoo_1.8-4.tar.gz'
Content type 'application/x-gzip' length 856279 bytes (836 KB)
==================================================
downloaded 836 KB

* installing *source* package ‘zoo’ ...
** package ‘zoo’ successfully unpacked and MD5 sums checked
** libs
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I../inst/include     -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-3.5.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c coredata.c -o coredata.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I../inst/include     -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-3.5.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c init.c -o init.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG -I../inst/include     -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-3.5.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c lag.c -o lag.o
gcc -std=gnu99 -shared -L/usr/lib/R/lib -Wl,-z,relro -o zoo.so coredata.o init.o lag.o -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/zoo/libs
** R
** demo
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (zoo)

The downloaded source packages are in
        ‘/tmp/downloaded_packages’
root@487d901bf794:/# exit
rob:~$

Standard images work just fine.

AndreaPi commented 5 years ago

Can you backtrace what/where it breaks?

Sure, if you tell me how to do that. I tried the rocker/rstudio image, on top of which the rocker/tidyverse is built:

docker run -d -p 8787:8787 -e PASSWORD=<mypwd> -e http_proxy=<myproxy> -e https_proxy=<myproxy> -e no_proxy=localhost --name rstudio rocker/rstudio

then I enter the RStudio GUI, and I have exactly the same problem:

> install.packages("janitor")
Warning in install.packages :
  unable to access index for repository https://cran.rstudio.com/src/contrib:
  cannot open URL 'https://cran.rstudio.com/src/contrib/PACKAGES'
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
Warning in install.packages :
  unable to access index for repository https://cran.rstudio.com/src/contrib:
  cannot open URL 'https://cran.rstudio.com/src/contrib/PACKAGES'
Warning in install.packages :
  package ‘janitor’ is not available (for R version 3.5.1)

it looks like you have a networking issue here, and it may well be cause by you and your proxy.

Nope. I can download Docker images to the server (which means that I correctly set the proxy on the remote server) and I can run R in Jupyter from a Docker container just fine (which means that I can correctly setup proxies in running containers). It's just rocker/rstudio which is giving me an headache.

Witness:

rob:~$ docker run --rm -ti r-base bash
root@487d901bf794:/# install.r zoo

What's install.r? When I run either the rocker/rstudioor the rocker/tidyverse container and I access the RStudio GUI through a browser, entering install.r zoo at the R console just throws a different error:

> install.r zoo
Error: unexpected symbol in "install.r zoo"

I also tried your way, and it still doesn't work:

$ docker run -it --rm -e PASSWORD=<mypwd> -e http_proxy=<myproxy> -e https_proxy=<myproxy> -e no_proxy=localhost --name rstudio rocker/rstudio bash
root@39b558f8e43e:/# install.r zoo
bash: install.r: command not found

cboettig commented 5 years ago

Hey @AndreaPi , thanks for the bug report! From your command, it looks like you're behind some kind of proxy and are passing some environmental variables related to that (e.g. -e http_proxy). I don't have a lot of experience working with proxy services, but are you sure this is getting configured correctly?

Can you start with a shell in the base image, e.g. docker run --rm -ti r-base bash and then see if you can connect by setting the proxy env vars manually in the shell first? (Also, sorry, rocker/rstudio only has the install2.r script, but note Dirk's example suggested testing this with r-base first).

AndreaPi commented 5 years ago

Hi @cboettig , you're welcome. Yes, I'm sure I'm configuring the environmental variables correctly because as I said before, with the same exact configuration I'm able to both:

access the Docker public repository from the server and download images
run Jupyter in a Docker container on the remote server, connect to Jupyter from my local machine in my web browser

Thus the issue seems to be with rocker. It looks like it's not passing the environmental variables correctly to RStudio.

Can you start with a shell in the base image, e.g. docker run --rm -ti r-base bash and then see if you can connect by setting the proxy env vars manually in the shell first? (Also, sorry, rocker/rstudio only has the install2.r script, but note Dirk's example suggested testing this with r-base first).

Ok, I was testing with rocker/rstudio because I need that to work (I don't need Docker to run R from the CLI on a remote server), but sure, if it helps you, here is the test you required:

$ docker run --rm -ti r-base bash
Unable to find image 'r-base:latest' locally
latest: Pulling from library/r-base
5fc5a8c3aca5: Pull complete
5d0bc8b1173b: Pull complete
3ca4b634b502: Pull complete
cf75621eeb8a: Pull complete
8e6c0d1a6765: Pull complete
adc1db090481: Pull complete
Digest: sha256:5f6cc040e5174a4ee9294b1afd0e4563d97bea033a01db6543555a5d7c92faa9
Status: Downloaded newer image for r-base:latest
root@8e0b008d6434:/# export http_proxy=<myproy>
root@8e0b008d6434:/# export https_proxy=<myproy>
root@8e0b008d6434:/# export no_proxy=localhost
root@8e0b008d6434:/# install.r zoo
Warning: unable to access index for repository https://cloud.r-project.org/src/contrib:
  cannot open URL 'https://cloud.r-project.org/src/contrib/PACKAGES'
Warning message:
package ‘zoo’ is not available (for R version 3.5.1)

Of course I doctored the proxy lines, but I used the right ones in the actual test.

If you need me to perform any other test, I'd be glad to.

eddelbuettel commented 5 years ago

Maybe see some of the help on eg StackOverflow or the help lists for proxies and CRAN. This is not a Rocker issue -- it's "merely" your networking. We cannot help with that, sadly. Sorry.

AndreaPi commented 5 years ago

Ok, I thought it was related to Rocker because I can access the Internet through the proxy from other containers, but if it's not, I'll ask on the RStudio community or Stack Overflow. Thanks

eddelbuettel commented 5 years ago

"The internet" or CRAN? Methinks you have a specific issue with the package index.

AndreaPi commented 5 years ago

"The Internet". For example, when building images, I'm able to use apt-get update inside a Dockerfile to update the list of Ubuntu packages:

Step 5/8 : RUN apt-get update
 ---> Running in 8fd4558fe713
Get:1 http://archive.ubuntu.com/ubuntu cosmic InRelease [242 kB]
Get:2 http://security.ubuntu.com/ubuntu cosmic-security InRelease [83.2 kB]
Get:3 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease [83.2 kB]
Get:4 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease [65.5 kB]
Get:5 http://security.ubuntu.com/ubuntu cosmic-security/universe amd64 Packages [26.2 kB]
Get:6 http://security.ubuntu.com/ubuntu cosmic-security/main amd64 Packages [42.1 kB]
Get:7 http://archive.ubuntu.com/ubuntu cosmic/multiverse amd64 Packages [195 kB]
Get:8 http://archive.ubuntu.com/ubuntu cosmic/restricted amd64 Packages [13.6 kB]
Get:9 http://archive.ubuntu.com/ubuntu cosmic/universe amd64 Packages [11.6 MB]
Get:10 http://archive.ubuntu.com/ubuntu cosmic/main amd64 Packages [1342 kB]
Get:11 http://archive.ubuntu.com/ubuntu cosmic-updates/universe amd64 Packages [35.9 kB]
Get:12 http://archive.ubuntu.com/ubuntu cosmic-updates/main amd64 Packages [62.4 kB]

I guess you're right: the problem could be with the CRAN package index, then.

eddelbuettel commented 5 years ago

Which gets us back to your specific issue of your proxy settings etc as they interact with CRAN. You only documented that you cannot update / install packages from Docker when at your company location.

cboettig commented 5 years ago

Updating issue title to reflect problem more precisely and closing this as stale.

chris-villarroel commented 4 years ago

You have to create the .Renviron file in you home directory with the proxy variables. http_proxy= https_proxy= ftp_proxy= rsync_proxy=

rocker-org / rocker-versioned

Cannot install packages from RStudio container when behind private firewall #110