search

miniclip / gonsul

A stand-alone alternative to git2consul
Other
97 stars 35 forks source link

readme

Gonsul - A Git to Consul tool, in Go

Docker image Checks and tests

This tool serves as an entry point for the Hashicorp's Consul KV store. Not only because Consul lacks of a built in audit mechanism, but also because having configurations managed in GIT, using a gitflow or a normal development-to-master flow is much friendly and familiar to any development team to manage configurations.

Downloads in releases page.

Important Notes

The master branch is being refactored. No (big) breaking changes will happen but codebase is under significant changes. Mainly adding tests, makefile, and code refactor/cleanup.

Please use the releases/downloads page. Instructions for development will be added soon!

How It Processes a Repository

Gonsul will (optionally) clone your repository into the filesystem. After, it will recursively parse all the files in the directory. Whenever Gonsul moves one level deep into a folder, the folder name is added as a Consul KV path part and as soon as it finds a file (either .json, .txt or .ini - or any other given in parameters) it will take the file name (without the extension) and append to the Consul KV path, making it a key and the file content is added as the value.

Example: Take this repository folder structure:

+-- prod
|   |
|   +-- website1
|   |   |-- config.json
|   |   |-- db-pass.txt
|   |
|   +-- app2
|       |-- config.json
|
+-- dev
    |
    +-- website1
    |   |-- config.json
    |
    +-- app2
        |-- config.json

All the files content would be inserted, respectively, into the following Consul KV paths:

Note 1: Gonsul makes the assumption that only .txt, .json and .ini files are to be treated as keys and their content as values for Consul. Any other files are disregarded. Also, .json files are treated and validated as valid JSON, and inserted into Consul pretty formatted.

Note 2: You can instruct Gonsul to expand the JSON files, so it will parse the structure, appending the JSON path to the previous folder structure, creating single Consul KV entries for each value. Caveats: Any arrays found are inserted into Consul a bracketed comma separated value string. More details on this on the flags description.

Some Features

Gonsul was built out of necessity and so it provides some features that makes our team more comfortable regarding security and consistency, so below are the most important features (TL;DR's in bold):

Available Flags

Below are all available command line flags for starting Gonsul. Flags may be specified on the command line, using environment variables (flag prefixed with "GONSUL", converted to upper case, and "-" changed to "", e.g. GONSUL_REPO_URL), or using a configuration file (--config=). The order of precedence for specifying flags is:

  1. Command line options
  2. Environment variables
  3. Configuration file
  4. Default values
--config=
--strategy=
--repo-url=
--repo-ssh-key=
--repo-ssh-user=
--repo-branch=
--repo-remote-name=
--repo-base-path=
--repo-root=
--consul-url=
--consul-acl=
--consul-base-path=
--log-level=
--expand-json=
--secrets-file=
--allow-deletes=
--poll-interval=
--input-ext=
--keep-ext=

Below is the full description for each individual command line flag.

--config

require: no example: --config=gonsul.conf

This specifies a file with configuration settings for the options described below. File syntax:

--strategy

require: yes example: --strategy=ONCE

This defines the mode that Gonsul should run at, having the following strategies:

NOTES: On both POLL and HOOK strategies, the application will gracefully terminate upon receiving a SIGINT signal, but it will obviously not under a SIGKILL in which you can end up with inconsistent data inside Consul KV cluster (in case you have more than 64 operations in progress, and Gonsul issued only the first transaction for example)

--repo-url

require: no example: --repo-url=git@github.com:githubtraining/hellogitworld.git

This flag will tell Gonsul where it should clone/checkout the repository from. This should be a fully qualified GIT url (either ssh, http or https). Please provide the full URL, with any ports, credentials or whatever you would normally use.

Note: If you do not provide this flag, Gonsul will still look at the filesystem --repo-root folder and try to process/parse the directory. This can be useful in case you want to delegate the repository cloning to a CI platform, such as Bamboo for example, removing the GIT responsibility from Gonsul and having it just process the files and sync them with Consul as normal.

--repo-ssh-key

require: no example: --repo-ssh-key=/home/example_user/.ssh/id_rsa

This is only required if the previous flag was given and it tells Gonsul where it can find the SSH key file to use in case we're connecting through SSH with a key.

--repo-ssh-user

require: no example: --repo-ssh-user=git

This is only required if the previous flag was given and it tell Gonsul what user we should use when connecting to an SSH GIT repository.

--repo-branch

require: no default: master example: --repo-branch=my_branch_name

This is the branch name that Gonsul should try to checkout.

--repo-remote-name

require: no default: origin example: --repo-remote-name=upstream_name

This is the name of the remote configured on the repository. We need it to properly trigger some PULL and CHECKOUT commands on the repository.

--repo-base-path

require: no example: --repo-base-path=configs/relative/path

This is the relative folder, inside our repository, that Gonsul should consider and parse as values to be synced with your Consul cluster. If no value is given, Gonsul assume that the root repository is to be considered as Consul parseable.

Note: Gonsul will only build the hierarchy path from this path onward. Given the example above, Gonsul will not try to look for a Consul KV path that starts with configs/relative/path but instead, it will from any deeper path from this folder down.

Note2: If this flag is not set, the below value of --repo-root will be used as the hierarchy path within Consul, which is normally not desired. Most of the times we should/must use this flag.

--repo-root

require: yes example: --repo-root=/home/user/gonsul/repo_dir

This is the absolute path where Gonsul should clone the repository in your OS filesystem. This is a required field, and when provided without providing the previous flag --repo-url, Gonsul will try to parse the given absolute path without using GIT. This could be useful if you want to use Gonsul as filesystem parser and Consul sync without doing any GIT operations.

Note: This value/path will be used as the the hierarchy path on Consul if no value is given on above --repo-base-path which in many cases is not intended. Most of the times, we should also use the flag above.

--consul-url

require: yes example: --consul-url=https://consul-cluster.example.com:8080

This is the Consul's REST API endpoint that Gonsul will call to make the required inserts, updates and deletes on the Consul's KV store. Please provide the full URL, with scheme and port if appropriate.

Note: Do not add the KV path, just the the scheme and the authority part of the URL.

--consul-acl

require: no example: --consul-acl=youracltokenhere

This is the Consul's access token that Gonsul will use when connecting to the cluster agent. This ACL must have read and write access to the same KV paths that are going to be replicated with the repository files.

--consul-base-path

require: no example: --consul-base-path=my/kv/base/path

This is the prefix for all generated keys that Gonsul will look at.

Note: Remember that this base path must not be mirrored in the repository, as it will be automatically appended. This is useful when the Consul cluster as all the KV paths segregated (namespaced) by teams or projects.

--log-level

require: no default: ERROR example: --log-level=DEBUG

This defines the logging level that Gonsul should run, having the following levels:

--expand-json

require: no default: false example: --expand-json=true

This will tell Gonsul how to treat JSON files. If true, Gonsul will traverse the JSON files and append the path to the previous folder/file hierarchy and create single entries in Consul KV for each value.

Note: Because Consul is a simple KV Store, where all values are strings, there are some caveats regarding the JSON file expanding. Some important ones are:

--expand-yaml

require: no default: false example: --expand-yaml=true

This will tell Gonsul how to treat YAML files. If true, Gonsul will traverse the YAML files and append the path to the previous folder/file hierarchy and create single entries in Consul KV for each value.

Note: Because Consul is a simple KV Store, where all values are strings, there are some caveats regarding the YAML file expanding. Some important ones are:

--secrets-file

require: no example: --secret-files=secrets.json

This is the location for a secrets file. You can either pass a relative path and Gonsul will look for it into the --repo-root path or you can pass an absolute path. If this value is passed and the file is found and checked as valid, Gonsul will try to do an on-the-fly search and replace any placeholders for the corresponding values. This is done using mustache template system.

This file must be a valid JSON map, where the keys are the placeholders and the values the actual secrets. An example secrets.json file could look like this:

{
   "FOO_DB_USER": "foo_username_foo1982",
   "FOO_DB_PASS": "foo_password_J5sXoEN",
   "BAR_TOKEN": "bar_token_J5sXoEN",
   "BAZ_KEY": "baz_key_J5sXoEN"
}

Note 1: All the replacement is done on-the-fly in memory, and apart from the original supplied secrets.json file, no secrets are written to disk. Note 2: The placeholders should follow the mustache triple curly braces {{{FOO_DB_USER}}}, that means "unescaped HTML charcaters" - basically takes the value as is.

--allow-deletes

require: no default: false example: --allow-deletes=true

This option sets Gonsul behaviour on how it should proceed in case some Consul deletes are to be made.

--poll-interval

require: no default: 60 example: --poll-interval=300

This is the number of seconds you want Gonsul to wait between checks on the repository when it is running in --strategy=POLL mode.

--input-ext

require: no default: json,txt,ini example: --input-ext=json,txt,ini,yaml

This is the file extensions that Gonsul should consider as inputs to populate our Consul. Please set each extension without the dot, and separate each extension with a comma.

--keep-ext

require: no default: false example: --keep-ext=true

Gonsul default behavior is to remove/trim the filextension from the filename when pushing files to consul k/v path, if this option is set to true gonsul will keep the file extension.

--timeout

require: no default: 5 example: --timeout=20

The number of seconds for the client to wait for a response from Consul

Gonsul Exit Codes

Whenever an error occurs, and Gonsul exits with a code other than 0, we try to return a meaningful code, such as:

Contributing

For notes on how to contribute check CONTRIBUTING.