How do we want to handle volumes and "persistance" versus "distribution"

[james-nesbitt]

This is a meta-issue, calling for all opinions. This image is a result of various other issues and PRs:

• https://github.com/wunderkraut/wundertools-dockerprototype/issues/64
• https://github.com/wunderkraut/wundertools-dockerprototype/pull/63

Persistant volumes, and named volumes make management of local containers much easier.

1. Using a named/persistant volume means that you can keep your db data even when removing/upgrading the db container, and it makes it much easier to identify voumes when using the docker volumes command.
2. Using a volume means that you can never commit changes in the volumes into image versions, which means that you can never distributed the volume contents as a part of an image.

Consider the following:

1. if we want to rely on production environments, that use docker-compose.yml syntax, with no builds/repo content, then we can not use anything from outside of an image as a part of our application:
   • source code, assets, db-data must all be inside an image.
   • these images can be versioned/tagged, and the versions can be the basis for snapshot/rollbacks actions during deployments
   • images can be exported, which can be the basis for bit-wise archiving.

[james-nesbitt]

@artursv and @aleksijohansson please comment.

[james-nesbitt]

It occurs to me that people may not know the difference between

$/> wundertools down

Which removes all containers and

$/> wundertools stop

Which simply stops them.

These are of course docker-compose semantics, not ours.

[james-nesbitt]

I am playing now with methods for rebuilding images from running containers and am already running into problems when not using named/global volumes. It is looking more and more like we will need to base deployment upgrades on scripts related to volumes.

[artursv]

We could rename the 'down' operation to something that tells the user that container is going to be destroyed completely - wundertools destroy for example (yes, I'm thinking vagrant again).

Having this and in the future - confirmation step before wundertools down - would probably eliminate the need for persistent volumes.

[aleksijohansson]

Edit: Removed stupid uneducated comment.

[aleksijohansson]

Edit: Removed stupid uneducated comment.

[hkirsman]

Regarding the down command. If it's docker specific then may be not change the name because of consistency. But if it destroys images then adding confirmation would be nice addon.

[aleksijohansson]

Forget what I said earlier, it seems I'm just way too behind on my Docker knowledge. It seems that there is already a solution for host independent data (volume) handling: https://docs.docker.com/engine/userguide/containers/dockervolumes/#mount-a-shared-storage-volume-as-a-data-volume

I think this no longer applies "if we want to rely on production environments, that use docker-compose.yml syntax, with no builds/repo content, then we can not use anything from outside of an image as a part of our application"

[aleksijohansson]

Here is a list of the volume plugins: https://github.com/docker/docker/blob/master/docs/extend/plugins.md#volume-plugins

[aleksijohansson]

One of the plugins is Flocker, which might be worth a look: https://clusterhq.com/flocker/introduction/

"Flocker is an open-source container data volume manager for your Dockerized applications.

By providing tools for data migrations, Flocker gives ops teams the tools they need to run containerized stateful services like databases in production.

Unlike a Docker data volume which is tied to a single server, a Flocker data volume, called a dataset, is portable and can be used with any container in your cluster.

Flocker manages Docker containers and data volumes together. When you use Flocker to manage your stateful microservice, your volumes will follow your containers when they move between different hosts in your cluster."

[aleksijohansson]

Seems ceph is not available with Flocker yet, but there is another plugin that works with ceph: https://github.com/contiv/volplugin

"volplugin controls Ceph RBD devices in a way that makes them easy to use for devs with docker, and flexible to configure for ops. Reference your volumes with docker from anywhere Ceph is available, and they are located and mounted. It's great with Compose and Swarm!"

[aleksijohansson]

For a more manual and scriptable option for managing volume data there are tools available: https://github.com/discordianfish/docker-backup https://github.com/paimpozhil/docker-volume-backup

Also from the guys behind Flocker there is this: https://clusterhq.com/dvol/

"As a developer, you spend a lot of time manually and automatically testing applications, and to do that, you use sample data in local databases. To make your life easier, we've created dvol, a utility that brings git-like functionality to Docker volumes. This means you can commit, reset and branch your development databases just like you already can with your code. dvol is available as an Apache 2.0 download."

[aleksijohansson]

My conclusion is that we might get the best of both worlds by using volumes with a volume plugin.

[james-nesbitt]

agreed. I will just check on hosting compatibility.

[james-nesbitt]

Regarding "down" and "stop", these are stock docker-compose operations, not our own functionality. I understand that the old paradigm may have interpreted "down" differently, but does it make sense to keep the old paradigm, when the new one is becoming an industry standard.

That said, we can easily perform our own down implementation, and ask for confirmation (or automatically snapshot)

Most likely we will stay with named-volume based data.

[james-nesbitt]

@operinko comments?