esciara commented 7 years ago

Hi,

This is more of a question than an issue. I used volumerize to backup some Docker Compose managed named container volumes and it works great. Now I have moved to Docker Swarm and I am running into problems.

One of them was that to get the right volume names, which I resolved by putting the volumerize container in the same stack.

The second pb I have, not resolved yet, is about stopping and starting containers. Since the container managed by Swarm have names that cannot be determined in advance, I cannot use the VOLUMERIZE_CONTAINERS variable.

Any clue on how to work around this?

blacklabelops commented 7 years ago

Docker-Compose can handle generic container names and fixed names. For VOLUMERIZE_CONTAINERS you need fixed container names.

Example:

services:
  crowd_postgresql:
    image: blacklabelops/crowd
    container_name: crowd

Gives the container the fixed name crowd even when using docker-compose.

You can't simply move volumes from one stack to the other if the docker-compose stacks use different compose project names. The default project name is the docker-compose folder but you can set the project name yourself with environment variables on you computer:

export COMPOSE_PROJECT_NAME=yourprojectname

Afterwards the volume names will be the same.

Usually I migrate the volumes from one stack to the other:

Create backup
Start new stack
Restore to new stack

Problem solved!

esciara commented 7 years ago

Thanks for the quick feedback.

Yep, that works with Docker-Compose... Doesn't with Docker Swarm.

Apart for container_name not being supported in Swarm mode, the containers created have a hash added to the end of the default naming. For instance, when using you confluence container, the result is: yourprojectname_confluence.1.gdfnxmsm9n0zbwbdsq1f66fdh.

And in any case, when stopping a container, Swarm mode automatically starts another container, with a different hash at the end of container name. The only way to make sure that the volume is not tempered with during the backup would be to stop the service, which means removing it... which means that the service cannot be restarted without being redeployed.

Am I correct?

Does it mean that Volumerize cannot really be used as it is today in Swarm mode? Or does it mean that it needs to be associated with a tool that does hot backup such as the docker volume plugin from Blockbridge?

blacklabelops commented 7 years ago

Interesting topic. Actually you could say that any Docker integrated plugin will beat any self-constructed image solution. Volumerize is a Docker wrapper around the best cli solution i could find: Duplicity. Duplicity itself does not have any functionality for Docker purposes.

You could say that Volumerize is the best multi-purpose and most importantly free solution to deal with the backup problem. Volumerize is a drop-in solution that can be used without installing and configuring anything. And even Blockbridge does not support as many storage backends as Duplicity.

Well yes, Duplicity is in need of a Discovery mechanism otherwise it will not be able to track containers simply by their names.

Example:

Container-To-Backup

$ docker run -d -e "VOLUMERIZE_ID=ConfluenceToBeBackuped" anycontainer

Afterwards Volumerize simply needs to be told which IDs to stop rather than looking for a specific container name.

Well this would be work in progress.

esciara commented 7 years ago

Interesting indeed... And I did not think about docker pause and docker unpause, which might solve the stop-start pb.

Gotta rush right now, but if volumerise could look for containers that would include the string yourprojectname_yourcontainername.* and pause-unpause them might solve the issue.

blacklabelops commented 7 years ago

docker pause and docker unpause are unreliable mechanisms. When you freeze an application that is writing something to a file your about to backup, well, i think there is a high chance your getting an unrestorable backup.

Only applications which are read-only can be freezed and later on secured. Otherwise there is always the possibility to corrupt data when you freeze in the middle of write operations. Especially when you try it with a database.

esciara commented 7 years ago

I think I got it.

The reason the swarm was restarting the containers was that the restart_policy was not set, hence using the default condition, which is any (Doh!). So for the start and stop of the container to work, it has first to be set to something else (either on-failure or none).

Then, instead of passing container names to Volumerize, service names should be passed. Volumerize would then find all the containers related to the service; stop them; back them up; restart them.

So the steps go on like this:

Check that the restart_policy condition is not set to any. If it is, throw an error and quit the process
Loop through the services declared through the VOLUMERIZE_CONTAINERS variable, find all containers, stop them
Do the backup
Start all above found containers

Should there be any pb in that sequence, an error should be logs and the process be set to fail. (QUESTION: does Volumerize currently do that with existing features, so that it can be minitored?_

The documentation should be as follows:

Docker Service Restarts

This image can stop and start all containers in a Docker service before and after backup. Docker services are specified using the environment variable VOLUMERIZE_SERVICES. Just enter their names in a empty space separated list.

Note: the restart_policy condition of the services have to be set to something other than any so that containers are note restarted automatically by Docker when stopped.

Example:

Docker container application with name application_service
Docker container application database with name application_database_service

Note: Needs the parameter -v /var/run/docker.sock:/var/run/docker.sock in order to be able to start and stop containers on the host.

Example:

$ docker run -d \
    --name volumerize \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v jenkins_volume:/source:ro \
    -v backup_volume:/backup \
    -e "VOLUMERIZE_SOURCE=/source" \
    -e "VOLUMERIZE_TARGET=file:///backup" \
    -e "VOLUMERIZE_SERVICES=application_service application_database_service" \
    blacklabelops/volumerize

The startup routine will be applied to the following scripts: backup, backupFull, restore and periodBackup.

Test the routine!

$ docker exec volumerize backup

blacklabelops commented 7 years ago

Currently Volumerize can work with the following Docker restart-policy:

no
on-failure
unless-stopped

It obviously cannot work with the following policy:

always

The default is:

no

Docker-Compose V3 additionally offers the deploy section, this is only supported by Docker Swarm. And lets not forget Kubernetes's pod restart policies or Rancher's cattle.

What I want to say is that Volumerize is not being able to be responsible for a specific restart policy of a specific technology. The user must understand that you cannot backup a container that restarts itself autonomously. And Volumerize, in its current state, cannot be responsible, when the user is not even aware if its container restart policies. Everybody using Volumerize should be aware of the basics of doing safe backups. I will redocument the restart section though.

I really like the idea of using service names but this is also no term supported by Kubernetes or Cattle. Therefore I would like to go to apply a specific ID environment variable on the target container and process that one.

All in all I think this calls for a swarm specific image of Volumerize that can work like that. Support Swarm features and terms. I would like to support this with extension points inside my scripts but I will not be able to implement and support this specific image by myself. I do not use Docker Swarm myself.

Monitoring: Volumerize uses Jobber as a periodic task scheduler. You can check the logs for successfull runs and failures. Also you can check the tasks state with the jobber listcommand.

esciara commented 7 years ago

Agree with what you say and your point is quite sound.

Will have to think about how to bring that forward. I will close this issue for now, and further comment as needed.

esciara commented 7 years ago

Actually, I found a solution, and it is so simple, I am asking whether it should not be considered to be added to this image.

For the usage, the solution would be only to add a environment variable SWARM_MODE, which when set to true changes the creation of the stopContainers and startContainers scripts.

Currently the code is this: https://github.com/blacklabelops/volumerize/blob/f8ed523d906a81da98add7fd9cb73e6a4a88a127/imagescripts/create_docker_scripts.sh#L21-L32

and it would be changed to this:

if [ -n "${VOLUMERIZE_CONTAINERS}" ]; then
  DOCKER_CONTAINERS=${VOLUMERIZE_CONTAINERS}
  for container in $DOCKER_CONTAINERS
  do
    if [ -n "${SWARM_MODE}" ] && [ "${SWARM_MODE}" = "true" ]; then
      cat >> ${VOLUMERIZE_SCRIPT_DIR}/stopContainers <<_EOF_
docker ps --filter "name=${container}." --format "{{.Names}}" > ${VOLUMERIZE_SCRIPT_DIR}/${container}_containers_list; cat ${VOLUMERIZE_SCRIPT_DIR}/${container}_containers_list | xargs -n1 docker stop $line
_EOF_
      cat >> ${VOLUMERIZE_SCRIPT_DIR}/startContainers <<_EOF_
cat ${VOLUMERIZE_SCRIPT_DIR}/${container}_containers_list | xargs -n1 docker start $line
_EOF_
    else
      cat >> ${VOLUMERIZE_SCRIPT_DIR}/stopContainers <<_EOF_
docker stop ${container}
_EOF_
      cat >> ${VOLUMERIZE_SCRIPT_DIR}/startContainers <<_EOF_
docker start ${container}
_EOF_
    fi
  done
fi

What do you say?

EDIT: Need to see whether writing code with a docker service update -d=false ${container} to start the containers is not better, since it means there is no need to store the names in ${container}_containers_list, but it also means it creates new containers...

blacklabelops commented 7 years ago

I really thank you providing me with the discovery docker command! Have you tested it?

esciara commented 7 years ago

Yes I have. Works a treat.

blacklabelops commented 7 years ago

I have tested the command in my environment. --filter "name=${container}." gives me multiple containers all containing the ${container}-string. Behaves a little bit arbitrary. Is that really what you want?

On the other side going through labels gives a more sound solution, e.g.

$ docker ps --filter "label=com.docker.compose.service=crucible_s3_backup" --format "{{.Names}}"

Returns the actual container name for a docker-compose service.

Is there also a way docker swarm defines label we can use?

You can check with docker inspect container_name for suitable swarm labels

esciara commented 7 years ago

Sorry for the late reply.

Yes, it behaves arbitrarily because Swarm will create containers to fulfil the service that has been created. The string will be of the format {stack_name}_{service_name_in_compose_file}.{replicate_number}.{hashtag}. When a replicate container dies for some reason, it will recreate the equivalent container and only change the {hashtag}.

In the code I suggested, ${container}={stack_name}_{service_name_in_compose_file}.

So the fact that the ${container}-string. behaves arbitrarily in normal. We need all running containers for a specific service on a stack to be stopped in order to backup its volumes, and we cannot know their full names in advance, but just that it has "{stack_name}_{service_name_in_compose_file}." in it. This is why I use the ${VOLUMERIZE_SCRIPT_DIR}/${container}_containers_list file to temporarily store the container name, to know which ones to start again.

$ docker ps --filter "label=com.docker.swarm.service.name=${stack_name}_${service_name_in_compose_file}" --format "{{.Names}}"

provides indeed a cleaner solution!

Berndinox commented 6 years ago

Is this merged?

blacklabelops / volumerize

Stop and Start Containers in Docker Swarm mode #2

Docker Service Restarts