Use a virtualenv to install python dependencies

rocker-org / binder

Template for RStudio on Binder / JupyterHub

GNU General Public License v2.0

69 stars 32 forks source link

Use a virtualenv to install python dependencies #3

Closed yuvipanda closed 7 years ago

yuvipanda commented 7 years ago

Isolates it from base image python dependencies. Mixing apt and pip for python dependencies always causes problems at some point :)
Allows users to install python packages as unprivileged users
Hack way around R apparently not picking up PATH from environment

yuvipanda commented 7 years ago

@cboettig this is the PR I was working on! Do I need to manually port this to the individual versioned dirs?

eddelbuettel commented 7 years ago

Mixing apt and pip ... always causes problems

You can do what we do for R: let apt deal with /usr/lib, and use direct CRAN installations only on /usr/local/lib (which is enforced via a distinct (first) .libPaths() entry).

I don't do much Python, but for the bits I need /usr/local/lib worked just fine. My $0.02.

yuvipanda commented 7 years ago

@cboettig do you have an example Makefile I can copy?

yuvipanda commented 7 years ago

@eddelbuettel yeah, that's the other option. However, a virtualenv is a more canonical solution than fiddling with paths ourselves manually in Python, so I'm going to leave it here.

cboettig commented 7 years ago

https://github.com/rocker-org/rocker-versioned/blob/master/verse/Makefile

As you can see there, I actually disabled sync there for older versions once things stabilized out a bit. As you can also see, I'm pretty terrible at writing Makefiles so feel free to deviate from the script.

eddelbuettel commented 7 years ago

a virtualenv is a more canonical solution

Opinions vary, and I generally avoid virtualenv (or packrat). Also, this is an R project here so ... (but then I don't play with binder so whatever Carl decides is cool by me). Just piping in from the sides...

yuvipanda commented 7 years ago

@cboettig Check out the makefile now. It uses Docker's ARG support to build multiple versions easily, but it might be too clever. I'm guessing this doesn't work with the way you build your images, since they're built on DockerHub...

yuvipanda commented 7 years ago

@cboettig I'm actually going to revert my last commit, and just manually copy them around for just now & move the makefile to a different PR...

yuvipanda commented 7 years ago

@cboettig ok, makefile is in https://github.com/rocker-org/binder/pull/5 as a separate PR now. I'll work on that later!

I think this PR is complete though!

cboettig commented 7 years ago

Great, thanks much!

Right, we wanted the transparency of autobuilds, though we do declare Docker ARGs now and again to make things easy for local builds: https://github.com/rocker-org/rocker-versioned/blob/master/r-ver/Dockerfile#L8-L9

Speaking of too clever though, DockerHub does have the notion of post-build hooks, which we currently have to create the semantic version tags https://github.com/rocker-org/rocker-versioned/blob/master/r-ver/post_push. Not sure if there's a hook-based or other similar solution that would let us work around duplicating Dockerfiles and changing the version tag, but I'm a simple man and go with what I know.

This all looks pretty good to me. We still have #2 dangling but the current solution seems reasonable for the time being.