Non-R system requirements

[wlandau]

holepunch is magical! I am using it for https://github.com/wlandau/learndrake, and it makes the workshop materials so much more accessible.

The setup was mostly smooth, but I hit a snag because of the dependency on TensorFlow. rocker/binder omits TensorFlow (and rightly so) so learndrake's own Dockerfile borrows heavily from rocker/tensorflow. It works now, but it took some micromanaging to go through apt-get and pip3 and make sure to invoke the correct user for each.

I an wondering if we can or should do anything in holepunch's API to abstract away this sort of thing. Maybe have arguments in holepunch::write_dockerfile() to supply apt and python dependencies? Or maybe a .holepunch.yml file that borrows from .travis.yml?

[karthik]

👋@wlandau! Thanks for the kind comments!

The setup was mostly smooth, but I hit a snag because of the dependency on TensorFlow. rocker/binder omits TensorFlow (and rightly so) so learndrake's own Dockerfile borrows heavily from rocker/tensorflow. It works now, but it took some micromanaging to go through apt-get and pip3 and make sure to invoke the correct user for each.

Supporting Tensorflow is very much on the radar. For now I'm focusing on getting the first version up before working on more complex use cases such as ML and geospatial.

I an wondering if we can or should do anything in holepunch's API to abstract away this sort of thing. Maybe have arguments in holepunch::write_dockerfile() to supply apt and python dependencies? Or maybe a .holepunch.yml file that borrows from .travis.yml?

I was waiting to introduce support for apt/pip in the next release but you nudged me earlier with this request 😀. I have now pushed support for write_apt() and write_requirements(). Both are as self explanatory as they sound. So your workflow can now be:

write_compendium_description
write_apt(apt_input = "
texlive-latex-base
texlive-latex-recommended
texlive-science
texlive-latex-extra
texlive-fonts-recommended
dvipng
ghostscript")

# and then 

write_requirements(requirements = "
matplotlib==3.*
seaborn==0.8.1
")

write_dockerfile(...)

Can you install the latest (0.1.13.9000) and let me know if that improves the workflow?

[karthik]

Also worth noting that the Rocker team is planning to support Tensorflow dailies soon so a end user might not have to mess too much with apt and requirements.

[karthik]

I spoke too soon. Having a Dockerfile will likely just override anything in apt and requirements. 😞

I'm hoping to introduce reference environments in a future release, where a downstream user will not have to build their own Dockerfile/Binder setup. They can point to someone else's binder [for compute environment] and their own repo [for code].

[wlandau]

Glad to hear you are working on easier configuration! I have added a new branch to learndrake for testing holepunch, and this is my latest attempt. I look forward to more tinkering.

[betatim]

@wlandau might be worth to look at using repo2docker directly which supports apt.txt and requirements.txt. This is the tool that powers mybinder.org and you can use it locally as well to build a docker image from your repo. A list of all the config files supported

If you install repo2docker directly from GitHub rebuilds of your repo that don't change the dependencies should be instant. This is a new feature for the upcoming release!

[wlandau]

Awesome! I will check it out when I get the chance.

[pat-s]

Just letting you know that the direct way via repo2docker works by providing a apt.txt file in combination with runtime.txt. Thanks @betatim! See giscience-fsu/daad_summerschool_hyperspectral for an example.

The downside is that the R version is not configurable right now and the current default is R 3.4.4.

[betatim]

https://github.com/jupyter/repo2docker/issues/661 is waiting for who ever wants to see being able to change R versions enough that they tackle it ;) The point being that mybinder.org and everything that goes into it might look like a professional effort but in reality there are two or three people (and their spare time) behind it. This means the bottleneck is time to discuss and come to consensus and then implement things -> all help even at a low level is welcome.

[nuest]

@karthik (as you likely know) The containerit R package creates the required RUN statements to install the apt packages needed by R packages as part of the Dockerfile, based on sysreqs. Wholepunch could do the same copying some code, or we could figure out how to expose the required functions from containerit.

With the new support for R versions in runtime.txt, did you consider not generating the Dockerfile but generating the different configuration files supported by repo2docker (install.R, runtime.txt, apt.txt) ?

[karthik]

With the new support for R versions in runtime.txt, did you consider not generating the Dockerfile but generating the different configuration files supported by repo2docker (install.R, runtime.txt, apt.txt) ?

Both options are currently possible in holepunch as outlined in the readme and vignettes. One can either go for the Dockerfile approach, or the install + runtime + apt approach. The downside of the latter is just the long installation time.

I could certainly fold the apt.txt into the Dockerfile if both are present.

[nuest]

Great, thanks for the clarification - I need to play around more with wholepunch obviously.

So what I'd suggest is to streamline something like this in a way that works for the given DESCRIPTION file but also for all required R packages:

> holepunch::write_apt(apt_input = sysreqs::sysreqs(desc = file.path(path.package(package = "openssl"), "DESCRIPTION"), platform="linux-x86_64-ubuntu-gcc"))
> readLines(".binder/apt.txt")
[1] "libssl-dev"             "libssl-dev"             "git-core"               "pandoc pandoc-citeproc" "pandoc pandoc-citeproc" "libsodium-dev"    

In case of write_dockerfile(..), sysreqs can deliver the actual command:

> sysreqs::sysreq_commands(desc = file.path(path.package(package = "openssl"), "DESCRIPTION"), platform="linux-x86_64-ubuntu-gcc")
[1] "export DEBIAN_FRONTEND=noninteractive; apt-get -y update && apt-get install -y libssl-dev git-core pandoc pandoc-citeproc libsodium-dev"