Small Go utility to:
Why a dedicated utility though? Can't I just do it with curl?
Yes you can - but we kept wanting to:
We did this at first with some custom Ruby scripts - but the pattern was apparent and could be applied to many other files as well.
This replaces our previous custom Ruby/shell scripts with a single Go binary we can use to get data in and out of Consul's KV store.
There's an introductory blog post available that shows how it can be used.
kvexpress in --key hosts --file /etc/consul-template/output/hosts.consul --length 100 --sorted=true
--file
and --key
are passed along with the command. Quit if they're not present - there are no safe defaults for those flags.stop
key - if it's there - stop and exit..compare
file - we will use this data from now on..last
file - if it's not there - create it..compare
and .last
files blank? If not - let's continue..compare
and .last
files - if they're different - continue..compare
file - if it's different - then let's update. This is to guard against it running on multiple server nodes that might have different .last
files.data
, and checksum
keys..compare
to .last
--exec
is passed - run that command.kvexpress out -k hosts -f /etc/hosts.consul -l 100 -e 'sudo pkill -HUP dnsmasq'
--file
and --key
are passed along with the command. Quit if they're not present - there are no safe defaults for those flags.stop
key - if it's there - stop and exit.data
and checksum
keys out of Consul.data
is long enough and the checksum
as computed on this side matches the checksum
key - then continue.data
to the passed --file
location.--exec
is passed - run that command.Build instructions are available below.
Debian packages for Ubuntu can be downloaded from packagecloud.
Additional binaries can be downloaded from the releases page.
A Chef cookbook is available as well.
Instructions are available here.
If you want to push a file to all nodes in your Consul cluster quickly - you can use Consul and kvexpress.
kvexpress in --prefix tmp --key config --length 5 --url https://git.io/vzz5P
This places the file and checksum in:
tmp/config/data
tmp/config/checksum
To get the file on every node:
consul exec "sudo kvexpress out --prefix tmp --key config --length 5 -f /etc/file.conf -e 'sudo restart something here'"
If you're not seeing it work the way you expect - you can always add --verbose
to the kvexpress command - that will add lots of additional output.
A detailed list of commands is available here.
How are keys organized in Consul's KV store to work with kvexpress?
Underneath a global prefix /kvexpress/
- each directory represents a specific file we are distributing through the KV store.
Each directory is named for the unique key and has the following keys underneath it:
data
- where the configuration file is stored.checksum
- where the SHA256 of the data is stored.For example - the hosts
file is arranged like this:
/kvexpress/hosts/data
/kvexpress/hosts/checksum
There is an optional stop
key - that if present - will cause all in
and out
processes to stop before writing anything. Allows us to freeze the automatic process if we need to.
All logs are sent to syslog and are tagged with kvexpress
. To enable debug logs, please export KVEXPRESS_DEBUG=1
Can be built with the standard go toolchain: go get -u -v github.com/DataDog/kvexpress
To build manually - clone the repo then: make deps && make
To run integration tests: make deps && make && make test
- it will spin up an empty Consul and kill it after the run.
Because we use user.Current()
- you can't cross compile this. If you want to build for Linux - you must build on Linux. Closed Issue
To install Consul - there are instructions here.
To launch an empty Consul instance: make consul
We love pull requests from anyone. Details are available here.
This project adheres to the Open Code of Conduct. By participating, you are expected to honor this code.