search

myENA / consul-backinator

Command line Consul backup and restore utility supporting KVs, ACLs and Queries
Mozilla Public License 2.0
226 stars 22 forks source link

First pass at docker integration. Results in image ~7MB in size. #8

Closed hekaldama closed 8 years ago

hekaldama commented 8 years ago

from #7 @leprechau

@hekaldama I'm not sure I understand the point of having a dockerfile for this type of utility. It also wouldn't ensure a stable build as it's not going to respect the glide configuration.

I am not sure this is the best approach, but I am currently PoCing running a 5 node consul cluster as a docker 1.12 service and I wasn't going to have the tasks in the service volume mount any host directories into the container. So I need to handle the -data-dir from consul in some way to recover from a disaster scenario. My thought was to just let the consul container use its own FS and use a service like consul-backinator to run backups to a mounted DFS. I would then like to use consul-backinator as a crond service on my swarm cluster so therefore need this utility inside of an image.

I realize the above is a little densely written and please let me know if I can clarify anything.

Thanks.

aaronhurt commented 8 years ago

@hekaldama I think I follow your reasoning but I'd like to make a few modifications. I'm going to take this patchset and apply it to a branch.

hekaldama commented 8 years ago

Cool, thanks.

aaronhurt commented 8 years ago

@hekaldama Please check the latest commit to the new dockerized branch (https://github.com/myENA/consul-backinator/commit/1f760e6e2ea7b16f88bd9bec9a8db1987107546e)

I updated the Makefile to be a bit more general purpose and tied it in with glide and gox for building. This ensures you will get a valid binary for the container even when executing the build on a non-Linux host. Additionally, I opted to base the image on alpine:latest as opposed to scratch to prevent the need to maintain an up-to-date certificate bundle in my own repository and allow people some customization of the image if they so choose.

hekaldama commented 8 years ago

Cool! Sounds good. Are you planning on pushing it to hub.docker.com at all?

aaronhurt commented 8 years ago

@hekaldama if it works for you I'll merge it to master.

hekaldama commented 8 years ago

Okay. Will be on the lookout for the branch when you push it.

hekaldama commented 8 years ago

Seeing it, testing...

hekaldama commented 8 years ago

Did you add + commit everything? Not seeing Dockerfile

aaronhurt commented 8 years ago

I moved it to the build directory along with the other build related scripts. You should be able to just type make docker to build it.

hekaldama commented 8 years ago

Seeing it, thanks.

Ran make docker then ran docker run --rm -it my-ena/consul-backinator which gave the output of:

standard_init_linux.go:175: exec user process caused "no such file or directory"

I think this is because you changed the compile to not make a static bin anymore as seen through:

docker run --rm -it --entrypoint ash \
  my-ena/consul-backinator -c 'ldd /usr/local/bin/consul-backinator'
hekaldama commented 8 years ago

Used my version with static binary in my swarm cluster as a service and it is performing backups now! Cool stuff. Thanks for the code. Just need the static bin probably in the image now.

aaronhurt commented 8 years ago

@hekaldama I know the problem. I'm building on macOS and golang always builds static bins when cross-compiling. I'll add the flags to enforce static even when building on Linux platforms.

aaronhurt commented 8 years ago

This should fix it b4404f2cd644a8c41dde2fa853affe530ca2b97b ... Thanks for the testing.

hekaldama commented 8 years ago

Testing now.

hekaldama commented 8 years ago

Hmm, still getting the same issue. I can look more tomorrow.

aaronhurt commented 8 years ago

Okay, latest commit to the dockerized branch should fix these issues. It just needed a few tweaks to the gox options. I tested it on an actual Linux host and it does appear to be working.

[ahurt@bld01.dev consul-backinator]$ uname -a
Linux bld01.dev.ena.net 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[ahurt@bld01.dev consul-backinator]$ make docker
Installing glide ... Installing from glide.lock ... Installing gox ... Building release ... Packaging ... done
docker build -t my-ena/consul-backinator -f build/docker .
Sending build context to Docker daemon 58.38 MB
Step 1 : FROM alpine:latest
Trying to pull repository docker.io/library/alpine ...
latest: Pulling from docker.io/library/alpine

117f30b7ae3d: Pull complete
Digest: sha256:02eb5cfe4b721495135728ab4aea87418fd4edbfbf83612130a81191f0b2aae3
Status: Downloaded newer image for docker.io/alpine:latest
 ---> 7d23b3ca3463
Step 2 : RUN apk add --no-cache --no-progress ca-certificates
 ---> Running in ceb3a634e7d7
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
(1/1) Installing ca-certificates (20160104-r4)
Executing busybox-1.24.2-r11.trigger
Executing ca-certificates-20160104-r4.trigger
OK: 5 MiB in 12 packages
 ---> 7e4da32146bf
Removing intermediate container ceb3a634e7d7
Step 3 : ADD dist/consul-backinator-*-amd64-linux/consul-backinator-* /usr/local/bin/consul-backinator
 ---> 7b29924d6bc0
Removing intermediate container c4b4b2347a89
Step 4 : ENTRYPOINT /usr/local/bin/consul-backinator
 ---> Running in 868be5f17b7f
 ---> 238be1cf87c5
Removing intermediate container 868be5f17b7f
Successfully built 238be1cf87c5
[ahurt@bld01.dev consul-backinator]$ docker run --rm --entrypoint /bin/sh -it my-ena/consul-backinator -c 'ldd /usr/local/bin/consul-backinator'
ldd: /usr/local/bin/consul-backinator: Not a valid dynamic program
[ahurt@bld01.dev consul-backinator]$ docker run --rm -it my-ena/consul-backinator
usage: consul-backinator [--version] [--help] <command> [<args>]

Available commands are:
    backup     Perform a backup operation
    dump       Dump a backup file
    restore    Perform a restore operation
aaronhurt commented 8 years ago

Merged dockerized branch to master after testing a make docker on Linux and macOS and rolled a new v1.1 release.

hekaldama commented 8 years ago

Cool thanks. Will look tomorrow.

On Sep 19, 2016 4:59 PM, "Aaron Hurt" notifications@github.com wrote:

Closed #8 https://github.com/myENA/consul-backinator/pull/8.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/myENA/consul-backinator/pull/8#event-794916881, or mute the thread https://github.com/notifications/unsubscribe-auth/AABDFboYDrTZcMeUMgUkuUIFWjzCVtnTks5qryHKgaJpZM4KAz2Q .