Hashicorp Vault HA cluster based on Raft Consensus Algorithm

Vault HA cluster is based on Raft Storage Backend announced tech preview on 1.2.0 (July 30th, 2019), introduced a beta on 1.3.0 (November 14th, 2019)) and promoted out of beta on 1.4.0 (April 7th, 2020)

The Raft storage backend is used to persist Vault's data. Unlike other storage backends, Raft storage does not operate from a single source of data. Instead all the nodes in a Vault cluster will have a replicated copy of Vault's data. Data gets replicated across the all the nodes via the Raft Consensus Algorithm.

• High Availability – the Raft storage backend supports high availability.
• HashiCorp Supported – the Raft storage backend is officially supported by HashiCorp.

Key features:

• Can be run with low consumption of costs or even just on AWS Free Tier
• No external dependencies like Consul, ETCD, database, etc for storing data
• No need additional provisioning tools like Ansible, Chief of Puppet, all based on clear Terraform
• Module fully independent with zero-external resources dependencies and just optional like ACM for HTTPS etc
• Provisioning based on CoreOS ignitions so very fast, declarative and predictable
• Fast and easily manual creating snapshots (backups) from Vault UI (thanks Raft implementations)
• Integrated auto backups of data by Amazon snapshots and fully configurable scheduler (false by default)
• Integrated optional auto-unseal with built-in AWS KMS provisioning, external AWS KMS or Transit secret backend by another Vault
• Vault Raft data is storing on separate EBS volumes independent from a root filesystem
• Easily increase and decrease the count of nodes without losing of data just by running terraform apply (with some downtime)
• The cluster can be running-up from just one node to N-nodes with proportional distribution by availability zones in a region
• No data lost happens even all instances will be terminated by mistake (just need to redeploy by Terraform for restoring a cluster)
• Possible to upgrade or downgrade a Vault version across all cluster without losing of data (with some downtime)
• Communication between peers (nodes) encrypted by TLS v1.2+ with certificate-based client authentication by RSA-2048* (bidirectional TLS encryption and authentication)
• Communication between cluster and ALB (Load Balancer) encrypted by TLS v1.2+
• Free Amazon certificate (ACM) can be assigned to ALB for client-server encryption
• By default, all nodes are hidden in private subnet and just one port on ALB accessible outside (best AWS security practice)
• Optional generation SSH pair (RSA-4096*) and assigned to nodes (not recommended, better to provisioning external SSH public key)
• Access to instances by SSH also can by provisioning Root CA* certificate and principals
• Provided assigning external own Root CA* for cases when Vault need secure communicate with internal infrastructure
• For better security provided support for disable_mlock=false by default
• Optional can enable assigning public network for all nodes. SSH and HTTPS port will be available publicly (for debugging and development only)
• Implemented debugging options with full audit of all configurations files and certificates which the cluster deploying on (for debugging and development only)

* looking here some limitations regarding AWS provisioning

Why?

Why not use a Kubernetes or other current cluster? For this, I can name a few reasons:

1. Independence. For creating infrastructure as a code by Terraform (for example the same cluster) we need storage for storing secret input parameters (passwords, IPs, private data) and outputs (tokens, endpoints, passwords). For this very convenient to use a Vault.
2. Stability. Cluster is an additional layer of abstraction across all EC2 instances. Much better using native methods of deploying.
3. Security. Vault might be storing very secret and sensitive data. Putting this data together with publicly available services may carry potential risks of leaks. In this case, we can deploy a cluster totally independent even in a separate AWS account with access to which is limited to a few people.
4. Lightweight. Sometimes we need a very lightweight and cheap Vault and at the same time very stable. E.g. just for auto-unseal another Vault.

IMPORTANT

• After Flatcar Container Linux release 2905.2.0 Vault cluster stop working due rkt deprecated, so all Vault module tags up to v0.1.8 stopped work, more #48. Please update module to latest version and check all the latest changes to compatibility with your configuration.
• From August 1, 2021 Flatcar Container Linux (Stable) by owner 075585003325 was removed from AMI public images and replaced by AMI Marketplace from owner 679593333241. This means all previous tags and Terraform code stopped work and need to update module or configure AMI manualy by ami_image. More about issue #60, #61
• For updating Vault module to version 0.2.x need some manual work, as there are some breaking changes, read more #61, #63
• From commit eaa9c48 the module supports Terraform versions newer than 0.12.x but still have backward compatibility with a Terraform v0.12.x but warns about unsupported source attribute, more #64. But from now we are announcing the deprecation of support for Terraform v0.12.x and support will be discontinued soon.

AWS Permissions

For deploying you need a list of permissions. For beginners might be difficult to set up minimal need permissions, so here the list wildcard for main actions. For professional or those who interesting for high-level security and granular permissions looking this AWS IAM Granular Permissions

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VaultHAProvisioning",
      "Effect": "Allow",
      "Action": [
        "ec2:*",
        "DLM:*",
        "elasticloadbalancing:*",
        "iam:*",
        "kms:*",
        "route53:*",
        "sts:GetCallerIdentity"
      ],
      "Resource": "*"
    }
  ]
}

Usage

IMPORTANT: The last code from master might need to temporary enable the option nat_enabled (access to external resources) at the first initialization since, during the creation of the cluster, the instance needs to get a docker image. An alternative could be placing the cluster on a public subnet

The module can be deployed with almost default values of variables. For more details of the default values looking here

provider "aws" {
  region = "us-east-1"
}

module "vault" {
  source = "github.com/binlab/terraform-aws-vault-ha-raft?ref=v0.1.8"

  cluster_name       = "vault-ha"
  node_instance_type = "t3a.small"
  autounseal         = true
  nat_enabled        = true
}

output "cluster_url" {
  value = module.vault_ha.cluster_url
}

Then run:

$ terraform init
$ terraform apply

After deploying the process you should see:

...
cluster_url = http://tf-vault-ha-alb-123456789.us-east-1.elb.amazonaws.com:443
$

Then just open URL in a browser and initialize the cluster

ATTENTION! Some resources cannot be covered by Amazon Free Tier or not Free usage and cost a money so after running this example should destroy all resources created previously by next command:

$ terraform destroy

HOW TO

1. Initializing newly created cluster
2. Raft manual snapshots (Init/Join/Backup/Restore)
3. AWS IAM Granular Permissions
4. Change AMI on worked cluster

Examples

1. Basic usage (Quick start)
2. Public SSH access to the instances by OpenSSH private key
3. Assigning CNAME and Route 53 Alias to Vault HA cluster
4. Adding public certificate and domain by ACM and Route53
5. Assigning module's VPC to external resources e.g. Bastion host
6. VPC Peering different networks e.g. RDS Database
7. Assigning Vault cluster to inside an already created (default) AWS VPC
8. Assigning external AWS KMS Key for internal Auto Unseal
9. Development and Debugging Sandbox

Troubleshooting

See separate page #troubleshooting

In case you encounter trouble that is not described in documentation or you cannot solve by your self please free to open an issue

TODO

• [x] Add examples of use with different cases #10
• [ ] Hosted module on Terraform Registry #13
• [ ] Add validation of input data in variables.tf
• [ ] Add support Fedora CoreOS as announced CoreOS Container Linux will reach its end of life on May 26, 2020 and will no longer receive updates.
• [x] Remove external dependency - VPC Module - #7
• [x] Add support for external AWS VPC - #4
• [ ] Add an option to configure preferred AWS availability zones - #36
• [ ] Add configuration for an external Vault Audit Device via syslog or socket
• [ ] Third-party plugins installation support
• [ ] Add optional opened HTTP port on ALB and setup redirect from HTTP to HTTPS. Canonical support
• [x] Disable NAT Gateway by default (for reducing costs consumptions and security improvement) - #27
• [ ] Option to disable Route 53 internal zone for (reducing costs consumptions)
• [ ] Add EFS storage support as a persistent Raft data storage
• [ ] Add option to disable creating an additional EFS (for reducing costs consumptions)
• [ ] Add option to store Raft data in a temporary memory (RAM) - paranoid mode
• [ ] Implement a provisioning internal Intermediate CA* for signing nodes certificates
• [ ] Implement scheduler backup of data by embedded snapshot operator and storing to S3 Bucket (reducing costs consumptions)
• [ ] Replace creating EC2 instances to an autoscaling group (might cost some limitation)
• [ ] Auto-provisioning a cluster on the first installation with storing Token and Unseal keys by GPG/PGP or Keybase
• [ ] Add support of OpenStack (OS) Terraform module
• [ ] Add support of Google Cloud Platform (GCP) Terraform module
• [ ] Add support of Microsoft Azure Terraform module
• [ ] Add support of AliCloud Terraform module
• [ ] Add support of Oracle Cloud (OCI) Terraform module
• [ ] Add support of Docker by Terraform (for local development and testing)
• [ ] Multi-regional support of cluster for super high availability
• [ ] Auto-deletion nodes in a cluster in time of decreasing count of nodes

* looking here some limitations regarding AWS provisioning

Limitations

• Because AWS strictly limiting the size of User Data file, we can't put into the ignition file a very big certificate and keys.

  User data is limited to 16 KB, in raw form, before it is base64-encoded. The size of a string of length n after base64-encoding is ceil(n/3)*4. source

  So in this case we need to select the size experimentally. For exactly knows size of the file, you can use debug mode

• Requirements block and versions.tf may not accurately display a real minimum version of providers. A declared versions ware just an installed in the time of development and testing of the module and can give guaranties of working with this or higher version. If you use older versions of modules for some reason and can give some guarantees of working with it, please create an issue for downscaling some version to minimal needed.

• According to the opened issue be careful with Tags settings, any changes after creating the tags may have a trigger effect on the change of value. Until the problem is closed by the Terraform team, a temporary workaround is applied and it is best to determine the tag names in advance

Requirements

Providers

Modules

No modules.

Resources

Inputs

[
  "0.0.0.0/0"
]

[
  "0.0.0.0/32"
]

[
  "vault-ha"
]

[
  "0.0.0.0/32"
]

[
  "sudo"
]

Outputs