unseal parameter for /sys/init

[jgiles]

Feature Request: Add an 'unseal' boolean parameter to /sys/init, defaulting to false. When passed, the Vault instance would unseal itself after initialization (before returning the unseal keys and the initial root token to the caller).

Use Case: Unsealing Vault at init time enables full automation of the following bootstrap workflow:

1. Create a Vault cluster.
2. Init + unseal one instance. Capture unseal keys + initial root token.
3. Provision core Vault configuration with initial root token by talking to unsealed instance.
4. Revoke initial root token.

That workflow takes the following steps (which previously needed to happen between steps 2 and 3 above) out-of-band:

1. Distribute (likely PGP-encrypted) unseal keys to their human owners.
2. Coordinate a threshold-satisfying number of human unseal key owners to unseal at least one Vault instance.

You would still need to perform the full unseal process on the other instances in an HA Vault setup, but only one instance is required to provision the cluster configuration (and in any case having multiple instances doesn't actually matter until an instance dies).

Security Implications: At init time, Vault already has all the information it needs to unseal itself (it has all the unseal keys).

The key security benefit of the above workflow is that it significantly shortens the lifetime and simplifies the handling of the initial root token. The root token can be held in the memory of the initialization client process for its full lifetime, and that lifetime can be limited to ~seconds.

If authentication mechanisms and policies are configured properly during the initial provisioning (using the root token), further Vault usage (and even policy-limited administration) can proceed without using a dangerous root token.

Current Workaround: You might be able to emulate the above workflow by initializing with a single unencrypted unseal key, unsealing with that key, provisioning the configuration, revoking the initial root token, and then rekeying to match the actual desired init setup. However, this process is significantly more complex.

[jgiles]

@jefferai thoughts on this? Paired with well-designed support in the Vault Terraform provider, this would permit a powerful pattern for automated Vault setup.

[jefferai]

I don't want to add this parameter because we've been thinking of making this the default anyways and we may also change the way that unseal tokens are returned to the caller.

[jgiles]

Ah, that's good to know. I think it makes sense for this to be the default behavior, but assumed backwards compatibility would win out.

What's the timeline for introducing this behavior? In what ways are you considering changing the unseal token return behavior?

We're trying to define our processes around this now, so any forward guidance would be much appreciated.

[jefferai]

What's the timeline for introducing this behavior? In what ways are you considering changing the unseal token return behavior?

Unknown. We've wanted to introduce another method for a long while but it's kept getting pushed back. Likely when we introduce it you won't get a root token back at all; you'll need unseal key holders to generate the initial root token, which makes oversight easier.

[jgiles]

Hmm, even if the Vault instance is unsealed at init-time, not returning a root token on init complicates the goal of "init+configure Vault automatically without human root-token handling". Enabling that pattern is the real thrust of this feature request.

[jefferai]

It's already dead easy to script up initial setup of Vault without a human ever seeing the root token. Not returning a root token would mean a bit more scripting work but also make it easier to provably ensure that no one human ever had control of the root token. Maybe I'm not really understanding what you're asking for.

[jgiles]

Ultimately, it would be great to have something in the Terraform provider like

resource vault_init "init" {
  pgp_keys = [
    "keybase:user1",
    "keybase:user2",
    "keybase:user3",
  ]
  secret_shares = 3
  secret_threshold = 2
}

which would do the initialization of a new Vault and capture the returned root token within the provider for use in authenticating the rest of the requests in the initial terraform apply. The encrypted unseal keys would be outputs of the resource.

Right now, this approach would require the Terraform provider to poll until operators unsealed a Vault instance (a bit unfortunate, and the motivation for this report). In a scenario where the root token needs to be generated separately and then provided as input it becomes much harder to do this with Terraform.

I know there are other challenges to structuring this as a Terraform resource (for example, it would be nice to cleanly support kicking off rekeys on seal config change, which might require some adjustments to the rekey process). The advantage is that the automation for secure setup would be codified for re-use in a way that integrates well both with the Terraform often used to deploy Vault instances and the Terraform provider for configuring already-running Vault instances.

[jefferai]

It seems like the super-easy workaround in both cases is for Terraform to do the pgp encrypting after the fact.

[jgiles]

Sure, that workaround would provide the desired behavior. It's a little unfortunate from a security perspective though... I suppose we are already placing a good deal of trust in the code/machine/operator running the Terraform provider, but holding all the raw unseal keys is a good deal scarier than holding a root token.

It also feels a tad hacky for an "endorsed" solution, though I guess that would be more for @paddycarver or other provider maintainers to judge.

I would still do that though if it were the best option available.

[jefferai]

It's not an endorsed solution. The only endorsed solution right now is for humans to initialize Vault. What is the reason you need Terraform to do it?

[jgiles]

I just meant that if it were to go into the Terraform provider it would certainly look like an at least semi-official solution. The reason for Terraform is that we want an automated setup mechanism which works well with our Terraform-created Vault cluster and our Terraform-codified Vault configs.

[paddycarver]

(Just chiming in to say that I'd very much like the Terraform provider to use endorsed solutions. In this case, if the endorsed solution is for humans to initialize Vault--which it sounds like it is--my preference would be for the Terraform provider to expect humans to have initialized Vault. If that changes--and it sounds like @jefferai has thoughts for that in the future--then I'd be interested in updating the provider to use the newly endorsed solution. But using non-endorsed solutions makes me afraid of backwards incompatible breaks being forced on the provider, and I want to avoid that whenever possible.)

[jefferai]

@paddycarver Humans initializing Vault is the endorsed solution because we don't have any other currently HC-supported solution.

If we wanted something natively supported by Terraform we'd have to figure out what that looks like.

[jgiles]

This user at least would love something natively supported by Terraform :-). Properly-implemented initialization automation in Terraform would provide significant security and usability benefits for Vault operators.

I probably should have brought @paddycarver into this discussion right from the beginning, but my intent when filing this feature request (an 'unseal' parameter for /sys/init) was to get an API in place that I could turn around and use in a proposal to https://github.com/terraform-providers/terraform-provider-vault for implementing a resource like that sketched out in https://github.com/hashicorp/vault/issues/4016#issuecomment-374046487.

The possible init changes described in https://github.com/hashicorp/vault/issues/4016#issuecomment-374029568 are related, but do not enable this goal. If and when they did happen they could probably be done in such a way to preserve an automation capability, so perhaps that API change should be treated as orthogonal?

The workaround in https://github.com/hashicorp/vault/issues/4016#issuecomment-374276563 seems unacceptable in an "endorsed" solution.

So... how do you feel about the original 'unseal' parameter proposal in light of the above? What other options might enable Terraform automation?

We're happy to help out with implementation both in Vault and in the Terraform provider on whatever solution is deemed acceptable. While assembling and test-driving our environment deployment, Vault initialization has been a major pain point.

[StephenWithPH]

This seems to cross over with https://github.com/terraform-providers/terraform-provider-vault/issues/122. Linking here so searchers can find the two issues.

[vishalnayak]

Issues that are not reproducible and/or not had any interaction for a long time are stale issues. Sometimes even the valid issues remain stale lacking traction either by the maintainers or the community. In order to provide faster responses and better engagement with the community, we strive to keep the issue tracker clean and the issue count low. In this regard, our current policy is to close stale issues after 30 days. Closed issues will still be indexed and available for future viewers. If users feel that the issue is still relevant but is wrongly closed, we encourage reopening them.

Please refer to our contributing guidelines for details on issue lifecycle.

[ChrisJBurns]

Was this ever taken further? Am currently looking at ways to automate the initialisation of Vault, we are currently having to write out own scripts that do this via the API, would much rather this be a terraform thing so that the entire lifecycle doesn't switch between automated and manual approach.

There's currently something like this where someone has taken it on themselves to plug the gap - although it doesn't perform an unseal. Would we be able to add init and unseal into the terraform-provider-vault? As far as I can see the API's allow for both init and unseal, as I have done this myself. Perhaps the last step is to add this bit into the terraform provider?

[aphorise]

Unseal itself after initialisation

• that's auto-unseal isnt it? - why should shamir self-unseal exceptionally on init? - how would the keys be verified unless some privileged operation is performed?

@jgiles - just curious is this still relevant for you?

[aphorise]

I feel that this request can be closed as auto-unseal already provides the requested functionality.

Also offering this explicitly for Shamir only (ie no autounseal) would then pose other issues to standby nodes where presumably all those would also be expected to auto-unseal on initial join (huge security gap IMO).

[heatherezell]

I'm going to go ahead and close this issue now. Please feel free to open a new one if needed - note, we accept bug reports and enhancement requests for "n-2" versions of Vault, which as of this writing would be 1.10+. Thanks!