This is a backend plugin to be used with Hashicorp Vault. This plugin generates either one-time (non-renewable) OAuth2 access tokens or service account keys with a given set of IAM roles bound to GCP resources for various GCP entities to authenticate with Vault.
Please note: We take Vault's security and our users' trust very seriously. If you believe you have found a security issue in Vault or with this plugin, please responsibly disclose by contacting us at security@hashicorp.com.
This is a Vault plugin and is meant to work with Vault. This guide assumes you have already installed Vault and have a basic understanding of how Vault works. Otherwise, first read this guide on how to get started with Vault.
If you are just interested in using this plugin with Vault, it is packaged with Vault and by default can be enabled by running:
$ vault secrets enable gcp
Success! Enabled the gcp secrets engine at: gcp/
If you are testing this plugin in an earlier version of Vault or want to test or use a custom build of the plugin, see the next section.
If you wish to work on this plugin, you'll first need Go installed on your machine (whichever version is required by Vault).
Make sure Go is properly installed, including setting up a GOPATH.
Clone this repository:
$ mkdir $GOPATH/src/github.com/hashicorp/vault-plugin-secrets-gcp`
$ cd $GOPATH/src/github.com/hashicorp/
$ git clone https://github.com/hashicorp/vault-plugin-secrets-gcp.git
or use go get github.com/hashicorp/vault-plugin-secrets-gcp
You can then download any required build tools by bootstrapping your environment:
$ make bootstrap
To compile a development version of this plugin, run make
or make dev
.
This will put the plugin binary in the bin
and $GOPATH/bin
folders. dev
mode will only generate the binary for your platform and is faster:
$ make
$ make dev
You will need a Vault instance running locally with a development plugin directory set up.
Start a local Vault server:
$ vault server -dev -dev-root-token-id=root -dev-plugin-dir="$GOPATH/vault-plugins"
Next you want Vault to use the newly built binary instead of the version bundled with Vault. You can use the scripted or manual methods depending on your preference.
Scripted method.
A Terraform project sets up the necessary gcp service account & credentials.
You can configure the project id via this environment variable: TF_VAR_GOOGLE_CLOUD_PROJECT_ID
or paste it in the terminal when prompted.
$ make setup-env
Source the environment variables generated by the setup project:
$ source ./bootstrap/terraform/local_environment_setup.sh
Enable the dev version of the plugin on your local Vault instance. You can specify the plugin name, development plugin directory and mount path or leave empty to use the default values.
$ make configure PLUGIN_NAME=vault-plugin-secrets-gcp PLUGIN_DIR=$GOPATH/vault-plugins PLUGIN_PATH=local-gcp
Manual method.
Save the name of your test project as an environment variable for reference:
$ export GOOGLE_CLOUD_PROJECT_ID=my-project # replace with your project ID
Do not use a production project. Use a dedicated project for testing.
Enable the IAM service on the project:
$ gcloud services enable --project "${GOOGLE_CLOUD_PROJECT_ID}" \
cloudresourcemanager.googleapis.com \
iam.googleapis.com
Create a testing service account:
$ gcloud iam service-accounts create vault-tester \
--display-name vault-tester \
--project "${GOOGLE_CLOUD_PROJECT_ID}"
Grant required test permissions:
$ gcloud projects add-iam-policy-binding "${GOOGLE_CLOUD_PROJECT_ID}" \
--member "serviceAccount:vault-tester@${GOOGLE_CLOUD_PROJECT_ID}.iam.gserviceaccount.com" \
--role "roles/owner"
$ gcloud projects add-iam-policy-binding "${GOOGLE_CLOUD_PROJECT_ID}" \
--member "serviceAccount:vault-tester@${GOOGLE_CLOUD_PROJECT_ID}.iam.gserviceaccount.com" \
--role "roles/iam.serviceAccountTokenCreator"
$ gcloud projects add-iam-policy-binding "${GOOGLE_CLOUD_PROJECT_ID}" \
--member "serviceAccount:vault-tester@${GOOGLE_CLOUD_PROJECT_ID}.iam.gserviceaccount.com" \
--role "roles/iam.serviceAccountKeyAdmin"
Note: these are overly broad permissions because the account needs a superset of all permissions it might grant. For this reason, it is strongly recommended that you have a dedicated project for running tests.
Download the service account key file to local disk:
$ gcloud iam service-accounts keys create vault-tester.json \
--iam-account "vault-tester@${GOOGLE_CLOUD_PROJECT_ID}.iam.gserviceaccount.com"
Export the credentials to an environment variable. You can set the env variable to either the path or the JSON itself, i.e.
$ export GOOGLE_TEST_CREDENTIALS="path/to/vault-tester.json"
Register the local plugin version
$ vault plugin register \
-sha256="$(shasum -a 256 "/path/to/plugin/dir/vault-plugin-secrets-gcp" | awk '{print $1}')" \
secret "vault-plugin-secrets-gcp"
Enable the GCP secrets engine
vault secrets enable --plugin-name="vault-plugin-secrets-gcp" --path="local-gcp" plugin
Write the configuration to allow connection to your google cloud project
vault write local-gcp/config credentials=@"$GOOGLE_TEST_CREDENTIALS"
To run the unit tests:
Run the unit tests:
$ make test
This plugin also has comprehensive acceptance tests covering most of the features of this secrets backend.
Warning: The acceptance tests create/destroy/modify real resources, which may incur real costs in some cases. In the presence of a bug, it is technically possible that broken backends could leave dangling data behind. Therefore, please run the acceptance tests at your own risk. At the very least, we recommend running them in their own private account for whatever backend you're testing.
To run the acceptance tests, you will need a GCP IAM service account with the appropriate permissions. You can use the steps explained in the setup section to configure one for you.
Save the name of your test project as an environment variable for reference:
$ export GOOGLE_CLOUD_PROJECT_ID=my-project # replace with your project ID
Run the acceptance tests:
$ make testacc
An IAM-enabled resource (under an arbitrary GCP service) supports the following three IAM methods:
getIamPolicy
setIamPolicy
testIamPermissions
In the case of this secrets engine, we need to call getIamPolicy
and setIamPolicy
on
an arbitrary resource under an arbitrary service, which would be difficult using
the generated Go google APIs. Instead,
we autogenerated a library, using the Google API Discovery Service
to find IAM-enabled resources and configure HTTP calls on arbitrary services/resources for IAM.
For each binding config resource block (with a resource name), we attempt to find the resource type based on the relative resource name and match it to a service config as seen in this autogenerated config file
To re-generate this file, run:
go generate github.com/hashicorp/vault-plugin-secrets-gcp/plugin/iamutil
In general, we try to make it so you can specify the resource as given in the HTTP API URL
(between base API URL and get/setIamPolicy suffix). For some possibly non-standard APIs, we have also
added exceptions to try to reach something more standard; a notable current example is the Google Cloud Storage API,
whose methods look like https://www.googleapis.com/storage/v1/b/bucket/o/object
where we accept either
b/bucket/o/object
or buckets/bucket/objects/object
as valid relative resource names.
If you are having trouble during role set creation with errors suggesting the resource format is invalid or API calls are failing for a resource you know exists, please report any issues you run into. It could be that the API is a non-standard form or we need to re-generate our config file.
See up-to-date engine docs and general API docs.