search

nginx-proxy / docker-gen

Generate files from docker container meta-data
MIT License
4.46k stars 605 forks source link
docker go

readme

docker-gen

Tests GitHub release Docker Image Size Docker stars Docker pulls

docker-gen is a file generator that renders templates using docker container meta-data.

It can be used to generate various kinds of files for:

Installation

There are three common ways to run docker-gen:

Host Install

Download the version you need, untar, and install to your PATH.

wget https://github.com/nginx-proxy/docker-gen/releases/download/0.12.0/docker-gen-linux-amd64-0.12.0.tar.gz
tar xvzf docker-gen-linux-amd64-0.12.0.tar.gz
./docker-gen

Bundled Container Install

Docker-gen can be bundled inside of a container along-side applications.

nginx-proxy/nginx-proxy trusted build is an example of running docker-gen within a container along-side nginx. jwilder/docker-register is an example of running docker-gen within a container to do service registration with etcd.

Separate Container Install

It can also be run as two separate containers using the nginx-proxy/docker-gen image, together with virtually any other image.

This is how you could run the official nginx image and have docker-gen generate a reverse proxy config in the same way that nginx-proxy works. You may want to do this to prevent having the docker socket bound to a publicly exposed container service.

Start nginx with a shared volume:

docker run -d -p 80:80 --name nginx -v /tmp/nginx:/etc/nginx/conf.d -t nginx

Fetch the template and start the docker-gen container with the shared volume:

mkdir -p /tmp/templates && cd /tmp/templates
curl -o nginx.tmpl https://raw.githubusercontent.com/nginx-proxy/docker-gen/main/templates/nginx.tmpl
docker run -d --name nginx-gen --volumes-from nginx \
   -v /var/run/docker.sock:/tmp/docker.sock:rw \
   -v /tmp/templates:/etc/docker-gen/templates \
   -t nginxproxy/docker-gen -notify-sighup nginx -watch -only-exposed /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf

Start a container, taking note of any Environment variables a container expects. See the top of a template for details.

docker run --env VIRTUAL_HOST='example.com' --env VIRTUAL_PORT=80 ...

Usage

$ docker-gen
Usage: docker-gen [options] template [dest]

Generate files from docker container meta-data

Options:
  -config value
      config files with template directives. Config files will be merged if this option is specified multiple times. (default [])
  -endpoint string
      docker api endpoint (tcp|unix://..). Default unix:///var/run/docker.sock
  -interval int
      notify command interval (secs)
  -keep-blank-lines
      keep blank lines in the output file
  -notify restart xyz
      run command after template is regenerated (e.g restart xyz)
  -notify-output
      log the output(stdout/stderr) of notify command
  -notify-sighup container-ID
      send HUP signal to container.
      Equivalent to 'docker kill -s HUP container-ID', or `-notify-container container-ID -notify-signal 1`.
      You can pass this option multiple times to send HUP to multiple containers.
  -notify-container container-ID
      send -notify-signal signal (defaults to 1 / HUP) to container.
      You can pass this option multiple times to notify multiple containers.
  -notify-filter key=value
      container filter for notification (e.g -notify-filter name=foo).
      You can pass this option multiple times to combine filters with AND.
      https://docs.docker.com/engine/reference/commandline/ps/#filter
  -notify-signal signal
      signal to send to the -notify-container and -notify-filter. -1 to call docker restart. Defaults to 1 aka. HUP.
      All available signals available on the dockerclient
      https://github.com/fsouza/go-dockerclient/blob/main/signal.go
  -only-exposed
      only include containers with exposed ports
  -only-published
      only include containers with published ports (implies -only-exposed)
  -include-stopped
      include stopped containers
  -tlscacert string
      path to TLS CA certificate file (default "~/.docker/machine/machines/default/ca.pem")
  -tlscert string
      path to TLS client certificate file (default "~/.docker/machine/machines/default/cert.pem")
  -tlskey string
      path to TLS client key file (default "~/.docker/machine/machines/default/key.pem")
  -tlsverify
      verify docker daemon's TLS certicate (default true)
  -version
      show version
  -watch
      watch for container changes
  -wait
      minimum (and/or maximum) duration to wait after each container change before triggering

Arguments:
  template - path to a template to generate
  dest - path to write the template. If not specfied, STDOUT is used

Environment Variables:
  DOCKER_HOST - default value for -endpoint
  DOCKER_CERT_PATH - directory path containing key.pem, cert.pm and ca.pem
  DOCKER_TLS_VERIFY - enable client TLS verification]

If no <dest> file is specified, the output is sent to stdout. Mainly useful for debugging.

Configuration file

Using the -config flag from above you can tell docker-gen to use the specified config file instead of command-line options. Multiple templates can be defined and they will be executed in the order that they appear in the config file.

An example configuration file, docker-gen.cfg can be found in the examples folder.

Configuration File Syntax

[[config]]
# Starts a configuration section

dest = "path/to/a/file"
# path to write the template. If not specfied, STDOUT is used

notifycmd = "/etc/init.d/foo reload"
# run command after template is regenerated (e.g restart xyz)

onlyexposed = true
# only include containers with exposed ports

template = "/path/to/a/template/file.tmpl"
# path to a template to generate

watch = true
# watch for container changes

wait = "500ms:2s"
# debounce changes with a min:max duration. Only applicable if watch = true

[config.NotifyContainers]
# Starts a notify container section

containername = 1
# container name followed by the signal to send

container_id = 1
# or the container id can be used followed by the signal to send

Putting it all together here is an example configuration file.

[[config]]
template = "/etc/nginx/nginx.conf.tmpl"
dest = "/etc/nginx/sites-available/default"
onlyexposed = true
notifycmd = "/etc/init.d/nginx reload"

[[config]]
template = "/etc/logrotate.conf.tmpl"
dest = "/etc/logrotate.d/docker"
watch = true

[[config]]
template = "/etc/docker-gen/templates/nginx.tmpl"
dest = "/etc/nginx/conf.d/default.conf"
watch = true
wait = "500ms:2s"

[config.NotifyContainers]
nginx = 1  # 1 is a signal number to be sent; here SIGHUP
e75a60548dc9 = 1  # a key can be either container name (nginx) or ID

Templating

The templates used by docker-gen are written using the Go text/template language. In addition to the built-in functions supplied by Go, docker-gen uses sprig and some additional functions to make it simpler (or possible) to generate your desired output. Some templates rely on environment variables within the container to make decisions on what to generate from the template.

Several templates may be parsed at once by using a semicolon (;) to delimit the template value. This can be used as a proxy for Golang's nested template functionality. In all cases, the main rendered template should go first.

[[config]]
template = "/etc/docker-gen/templates/nginx.tmpl;/etc/docker-gen/templates/header.tmpl"
dest = "/etc/nginx/conf.d/default.conf"
watch = true
wait = "500ms:2s"

Emit Structure

Within the templates, the object emitted by docker-gen will be a structure consisting of following Go structs:

type RuntimeContainer struct {
    ID           string
    Created      time.Time
    Addresses    []Address
    Networks     []Network
    Gateway      string
    Name         string
    Hostname     string
    Image        DockerImage
    Env          map[string]string
    Volumes      map[string]Volume
    Node         SwarmNode
    Labels       map[string]string
    IP           string
    IP6LinkLocal string
    IP6Global    string
    Mounts       []Mount
    State        State
}

type Address struct {
    IP           string
    IP6LinkLocal string
    IP6Global    string
    Port         string
    HostPort     string
    Proto        string
    HostIP       string
}

type Network struct {
    IP                  string
    Name                string
    Gateway             string
    EndpointID          string
    IPv6Gateway         string
    GlobalIPv6Address   string
    MacAddress          string
    GlobalIPv6PrefixLen int
    IPPrefixLen         int
    Internal            bool
}

type DockerImage struct {
    Registry   string
    Repository string
    Tag        string
}

type Mount struct {
  Name        string
  Source      string
  Destination string
  Driver      string
  Mode        string
  RW          bool
}

type Volume struct {
    Path      string
    HostPath  string
    ReadWrite bool
}

type SwarmNode struct {
    ID      string
    Name    string
    Address Address
}

type State struct {
    Running bool
    Health  Health
}

type Health struct {
    Status string
}

// Accessible from the root in templates as .Docker
type Docker struct {
    Name                 string
    NumContainers        int
    NumImages            int
    Version              string
    ApiVersion           string
    GoVersion            string
    OperatingSystem      string
    Architecture         string
    CurrentContainerID   string
}

// Host environment variables accessible from root in templates as .Env

For example, this is a JSON version of an emitted RuntimeContainer struct:

{
  "ID": "71e9768075836eb38557adcfc71a207386a0c597dbeda240cf905df79b18cebf",
  "Addresses": [
    {
      "IP": "172.17.0.4",
      "Port": "22",
      "Proto": "tcp",
      "HostIP": "192.168.10.24",
      "HostPort": "2222"
    }
  ],
  "Gateway": "172.17.42.1",
  "Node": {
    "ID": "I2VY:P7PF:TZD5:PGWB:QTI7:QDSP:C5UD:DYKR:XKKK:TRG2:M2BL:DFUN",
    "Name": "docker-test",
    "Address": {
      "IP": "192.168.10.24"
    }
  },
  "Labels": {
    "operatingsystem": "Ubuntu 14.04.2 LTS",
    "storagedriver": "devicemapper",
    "anything_foo": "something_bar"
  },
  "IP": "172.17.0.4",
  "Name": "docker_register",
  "Hostname": "71e976807583",
  "Image": {
    "Registry": "jwilder",
    "Repository": "docker-register"
  },
  "Env": {
    "ETCD_HOST": "172.17.42.1:4001",
    "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "DOCKER_HOST": "unix:///var/run/docker.sock",
    "HOST_IP": "172.17.42.1"
  },
  "Volumes": {
    "/mnt": {
      "Path": "/mnt",
      "HostPath": "/Users/joebob/tmp",
      "ReadWrite": true
    }
  }
}

Functions

Sprig functions that have the same name as docker-gen function (but different behaviour) are made available with the sprig prefix:

Some functions are aliases for Go's strings package functions:

Those have been aliased to Sprig functions with the same behaviour as the original docker-gen function:

Examples

NGINX Reverse Proxy Config

nginxproxy/nginx-proxy trusted build.

Start nginx-proxy:

docker run -d -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock -t nginxproxy/nginx-proxy

Then start containers with a VIRTUAL_HOST (and the VIRTUAL_PORT if more than one port is exposed) env variable:

docker run -e VIRTUAL_HOST=foo.bar.com -e VIRTUAL_PORT=80 -t ...

If you wanted to run docker-gen directly on the host, you could do it with:

docker-gen -only-published -watch -notify "/etc/init.d/nginx reload" templates/nginx.tmpl /etc/nginx/sites-enabled/default

Fluentd Log Management

This template generate a fluentd.conf file used by fluentd. It would then ship log files off the host.

docker-gen -watch -notify "restart fluentd" templates/fluentd.tmpl /etc/fluent/fluent.conf

Service Discovery in Etcd

This template is an example of generating a script that is then executed. This template generates a python script that is then executed which register containers in Etcd using its HTTP API.

docker-gen -notify "/bin/bash /tmp/etcd.sh" -interval 10 templates/etcd.tmpl /tmp/etcd.sh

Development

This project uses Go Modules for managing 3rd party dependencies. This means that at least go 1.11 is required.

For go 1.11 and go 1.12 it is additionally required to manually enable support by setting GO111MODULE=on. For later versions, this is not required.

git clone <your fork>
cd <your fork>
make get-deps
make

License

MIT