"Docker" vs "virtual environments" and "pipenv"

[dyerlytle]

When I was at USGS we used Python virtual environments. I see that Ryan recommends Docker. There is also pipenv to consider. What are the advantages and disadvantages of the various environment packaging systems and which can be used together and which would be better for our situation? Perhaps we can talk about this with the group.

It looks like Docker is more like a virtual machine than just a Python virtual environment. Is there overhead associated with the use of Docker images?

[astrobokonon]

Python virtual environments

For this, I'm grouping virtual environments (ostensibly managed via conda) along with pipenv since they're fairly similar. I'll edit this as needed.

Pros

• Good for quickly developing new stuff (and adding in a new package if needed)
• Runs on actual OS so no virtualization overhead or permissions concerns

Cons

• Hard to sync up without a lot of diligence; easy for developer A to get an environment that morphs into something completely different than developer B
• Must simultaneously manage OS and environment updates and interactions
  • In general, I find updating these things to be a chore. Anecdotally, I've never seen a well maintained and up-to-date virtual environment
• Installing new packages can be confusing; should I install it via conda install or pip install or build from source?
  • What about different conda repositories? anaconda vs. conda-forge vs. bioconda is the one that immediately comes to mind. There are already big version disparities for even the essential tools like pip between anaconda (pip version 10.0.1) and conda-forge (pip version 18.0).
  • 
• You can easily screw up and install packages into your OS python version rather than your virtual environment, or the wrong virtual environment. Recovering from the latter might be difficult if it triggered a bunch of other updates.

[astrobokonon]

Docker

I'm using "docker" below as a standard/stand-in term much like one can use "xerox" as a verb for "photocopy." Any virtualization solution could be used, such as Kubernetes (which still supports docker formats) or really any kind of Linux container platform.

Pros

• If you make an image once, you can distribute that exact same image everywhere that's needed elsewhere on the network
  • Build once, distribute everywhere
  • Fault tolerance is high, since the container could be started pretty much anywhere that docker (or whatever container system) is installed on the OS
• OS overhead is low since it's not a full-up virtual machine. Shares the basic kernel with the OS
• Docker is like an onion (or an ogre) since it has layers
  • 
  • Start with a base image, then add stuff to it; if you have additional requirements, then add another layer and it'll still use the same images up until that point

Cons

• If you're running a multi-user macOS machine, docker is god damn awful to interact with for everyone but the user that installed it. Thanks, Apple.
• Permissions can be harder to manage, since docker (and most containers) actually run as root by default; can be changed with effort/planning
• Needs adjustment of workflows to use them for actual development; more of a cultural consideration
  • Definitions/specifications -> Image(s) -> Container
    • Containers are completely distinct entities so this can be confusing at first. You can map local file system stuff into a container only, not an image
    • Containers are best be thought of as ephemeral and they (and their contents) are either deleted or just unavailable when the container is stopped. It's easy to get data out of them, or mount a local directory for storage that persists, but you need a plan for such things

[astrobokonon]

Summary

A hybrid approach might make the most sense?

conda or pipenv type environments might make the most sense for developers locally, but CI setups (like travis) basically use docker under the hood for builds so that's a strong consideration to bring docker into the mix earlier.