Versioning secrets

[wpg4665]

Would it be possible to implement something akin to versioned secrets? The intended usage would be for replacing secrets with new ones.

Example:

 Service X password -> Store password into Vault.
     vault http API access: http://127.0.0.1:8200/v1/secret/foo/v1
     which is also equivalent to: http://127.0.0.1:8200/v1/secret/foo/latest

Now, assume there's a policy that requires password rotating, so User Y happily goes along and changes the password, and updates the password in Vault

Service X NEW password -> Stored into Vault
    vault http API access: http://127.0.0.1:8200/v1/secret/foo/v2
    which is now equivalent to: http://127.0.0.1:8200/v1/secret/foo/latest
    /latest no longer points to /v1

BUT, and here's what I'm trying to solve, User Y didn't change the password on Service X (or changed it incorrectly); we now have the capability to retroactively, either perform a revert, so /latest goes back to pointing to /v1, or perform a lookup directly on /v1 to see what it was. Otherwise, access to Service X could likely be lost!

Is this something that's even remotely on the roadmap internally? Would it be a functionlity that you'd prefer to come from a PR? I'm not proficient in Go, although I could certainly try to contribute something along these lines.

[jefferai]

It's not on the roadmap currently, although not because we're not interested.

Part of the problem is avoiding key sprawl in Consul/etcd/ZK, because this would very likely massively increase the number of keys and they can't handle that very well. The alternative is to encode the versions within the values, which is better key-wise, but can also lead to the values getting very large over time -- something else these backends tend not to deal with very well.

At the moment, you could implement your own versioning mechanism using keys, and I think some have, but we haven't decided how to solve this properly yet. It may be as simple as making it tunable whether or not values are stored per-mount (but then what happens if you want to change that?), or it could even be allowing values to be versioned but leaving it up to the creator of an individual key whether or not to actually turn versioning on for that key.

[wpg4665]

Ok, that's awesome to hear! I think possibly a tunable expiration_date, or max_last_keys could be reasonable! Yeah, I was thinking about a customized frontend could also be an option, and as long as I always used that version-aware frontend, it could give me similar functionality. I just thought that it would be really nifty to have on the backend by default =)

[jefferai]

It would definitely be nifty. We just need to find a decent way to do it that won't have a high probability of blowing up people's storage. :-)

[clarkbreyman-yammer]

Could this be done with a custom backend - essentially a façade that loops back to another provider after performing a mapping on the secret or secret key?

[clarkbreyman-yammer]

Thinking more - mapping secret data to a history of the last N versions and filtering in the versioning backend might make more sense. We might want to keep a single backup of the whole copy in event of failure/corruption on update.

[kevinrbabcock]

@jefferai Why would versioning massively increase the number of keys? I can't think of a reason to store more than 2-3 versions of a particular key. Storing exactly two versions would dramatically simplify our workflow. Storing a few versions certainly increases storage utilization, but not by an order of magnitude.

A simple restriction on the number of versions stored a la HBase would solve the problem easily. It doesn't have to be configurable.

[jefferai]

@kevinrbabcock You can't think of a reason to store more than 2-3 versions, but that's you. Many people want to store an arbitrary number of versions over an arbitrary length of time. Having artificial cutoffs doesn't seem like the right approach, especially down in the 2-3 range.

[kevinrbabcock]

I agree others have different use cases from ours. My suggestion provides a way to satisfy the use cases that need 2-3 versions. If those use cases cover 80% of the people who need versioning, you've made a big improvement to the product without having to solve the storage problem. I believe that simplifies the implementation a lot.

The person who opened this ticket has the same use case we do: rotating secrets. No one has posted here a request for arbitrary versions.

[jefferai]

Agreed, nobody has communicated such a need on this ticket.

[bkrodgers]

I'd agree with both sides there -- yes, infinite versioning would be nice. However, if the technical constraints of key explosion are what blocks us from doing that, I think most people would be very happy having some versioning of some sort, with a cap imposed. 2-3 might be low, but something like 5 or 10 (perhaps configurable) would solve what most people are concerned with here. I'd hate to see this delayed indefinitely while we come up with a solution for infinite versioning, when a limited history would still be extremely valuable.

In addition to rotation, there's also accidental updates. For example, I've already had a user blow away some data because they didn't understand that vault write <existing-path> key1=value1 will blow away any keys other than key1. Obviously that was user error (or you can call it a training error) in that they didn't understand that a path is a single unit of KV pairs, and that there's no "update" function built into the CLI. But we are very concerned about the risk of losing secrets this way, or via any other form of inadvertent overwriting.

Yes, there are ways to manage this, such as training users to manually back up secrets to a -bak path before they write, and putting processes in place where users normally have read-only access with on-demand elevation. But still, Versioning would provide us with a very valuable safety net for such situations.

[richfromm]

I agree that if the worry is that infinite versioning will blow things up, don't allow a max that is infinite. Have some finite config value, with a sensible default. Sure, there's nothing stopping people from using MAX_INT, but appropriate documentation ought to be sufficient to warn people against that.

In addition to rotation, there's also accidental updates. For example, I've already had a user blow away some data because they didn't understand that vault write <existing-path> key1=value1 will blow away any keys other than key1.

I did that myself during initial testing before I more fully understood how vault works. The decision that I made was to define away the problem by adjusting your schema so that only one value is stored at each path, with the fixed key value, and your paths have one more level of hierarchy. So instead of:

vault write secret/some/path/foo key1=value1

You have:

vault write secret/some/path/foo/key1 value=value1

It's slightly less efficient, but IMHO there's too much of a risk of screwing up and losing data without it. Built in versioning would help somewhat, but it would still be better to avoid a problem rather than have to recover from it.

[ekristen]

Arbitrary limits tend to cause problems. I caution enforcing random limits for the sake of limits.

https://en.m.wikipedia.org/wiki/Zero_one_infinity_rule

Just my two cents.

Sent from my iPhone

On Dec 22, 2016, at 15:00, Rich Fromm notifications@github.com wrote:

I agree that if the worry is that infinite versioning will blow things up, don't allow a max that is infinite. Have some finite config value, with a sensible default. Sure, there's nothing stopping people from using MAX_INT, but appropriate documentation ought to be sufficient to warn people against that.

In addition to rotation, there's also accidental updates. For example, I've already had a user blow away some data because they didn't understand that vault write key1=value1 will blow away any keys other than key1.

I did that myself during initial testing before I more fully understood how vault works. The decision that I made was to define away the problem by adjusting your schema so that only one value is stored at each path, with the fixed key value, and your paths have one more level of hierarchy. So instead of:

vault write secret/some/path/foo key1=value1

You have:

vault write secret/some/path/foo/key1 value=value1

It's slightly less efficient, but IMHO there's too much of a risk of screwing up and losing data without it. Built in versioning would help somewhat, but it would still be better to avoid a problem rather than have to recover from it.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[wpg4665]

But that shouldn't stop anyone from setting "sane defaults"...would at least prevent the user from doing something bad...unless they REALLY meant to!

On December 22, 2016 at 21:49:59, Erik Kristensen (notifications@github.com) wrote:

Arbitrary limits tend to cause problems. I caution enforcing random limits for the sake of limits.

https://en.m.wikipedia.org/wiki/Zero_one_infinity_rule

Just my two cents.

Sent from my iPhone

On Dec 22, 2016, at 15:00, Rich Fromm notifications@github.com wrote:

I agree that if the worry is that infinite versioning will blow things up, don't allow a max that is infinite. Have some finite config value, with a sensible default. Sure, there's nothing stopping people from using MAX_INT, but appropriate documentation ought to be sufficient to warn people against that.

In addition to rotation, there's also accidental updates. For example, I've already had a user blow away some data because they didn't understand that vault write key1=value1 will blow away any keys other than key1.

I did that myself during initial testing before I more fully understood how vault works. The decision that I made was to define away the problem by adjusting your schema so that only one value is stored at each path, with the fixed key value, and your paths have one more level of hierarchy. So instead of:

vault write secret/some/path/foo key1=value1

You have:

vault write secret/some/path/foo/key1 value=value1

It's slightly less efficient, but IMHO there's too much of a risk of screwing up and losing data without it. Built in versioning would help somewhat, but it would still be better to avoid a problem rather than have to recover from it.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hashicorp/vault/issues/1364#issuecomment-268932666, or mute the thread https://github.com/notifications/unsubscribe-auth/AAl7QOUUqTdNu2otjU8kJhEOSVl3VcKGks5rKzbXgaJpZM4IVk7c .

[richfromm]

Perhaps I wasn't clear. I wasn't suggesting that vault set an arbitrary limit. I was suggesting that if versioning is enabled, the user needs to choose and set some limit as to how many versions. And that the consequences be clearly communicated, and if this option is not literally required, then the default value be something sane.

Although I was also making the point that it might be better to require whatever limit is chosen be some actual number, and not special case some setting that allows for truly infinite versions. Is that what you're disagreeing with? That the user should have the option of choosing infinite?

[rmharrison]

NB: Lyft's Confidant supports versioned secrets, see 'history view' in docs. Vendor-locked to AWS by design + requires python (https://github.com/lyft/confidant/issues/17), so YMMV.

[StyleT]

I would say that this feature is critical for Generic Secret Backend in terms of audit (audit backend can't show you history of some particular secret) & allows you to revert accidental changes.

[JonathanAtKT]

Thanks for the recommendation - I'll check out "Confidant" soon.

Here are two secret versioning use cases that my company (large online retailer using primarily Jenkins-powered CICD pipeline) has:

• "Zero downtime" cluster deployments: We push out a new credential, and the members of a cluster pick it up over the next few minutes/hours. During the whole time, both credentials are accepted. After all members of the cluster are using the second credentials, the first credential is dropped. (Historically, this is similar to enterprise support for rotated PGP keys: to avoid "missed batches" you'd import a partner's new key into the keychain and leave the partner's old key until they stopped using it.)

• Historical encrypted data: if we're archiving encrypted data, we need to archive the keys for that data if we're also rotating the encryption keys over time

[JonathanAtKT]

Hi - I was just in a concall with Hashicorp and I heard "the transit backend" solves these use cases.
https://www.vaultproject.io/docs/secrets/transit/ https://www.vaultproject.io/api/secret/transit/index.html

Essentially, an extra "vault:v1:" header (e.g.,) gets prepended to the normal ciphertext and the backend figures out which key to actually use to decipher ciphertext.

[jefferai]

@JonathanAtKT Yes, your use cases can be solved via transit -- you can rotate the key version and eventually to disable the old data set the min_decyption_version on the key to a later key version. But, if needed, an admin can set the min version back to something older to decrypt an older piece of data.

It's different from the strict versioning support for the K/V store that others are asking for in here but I'm glad it works for your use case!

[StyleT]

Hi! Is there any updates here? I have a need to setup secrets storage for human use in a company & vault fits perfectly at first glance, as it's already in use, in case this feature will be implemented. I don't think that it's a good idea to have a secrets storage which is editable by anyone in a team without "history view".

[odesskij]

any updates on this?

[jefferai]

None currently.

[gw4thanize]

Secret versioning would be a critical feature. Is it possible to add it to the roadmap?

Thanks

[jefferai]

We have some vague plans but I can't give any timelines at this point.

[marcofvera]

👍 for this feature

[Tensho]

👍 for this feature

[aaronlifton]

👍

[msrai]

👍

[jefferai]

Note from Admin: I deleted John's comment -- not because he's plugging CyberArk's Conjur, but because he failed to identify himself as being a senior product marketing manager for CyberArk.

John, we are fine with healthy competition, but if you want to continue to post here, you need to disclose that you are not an impartial entity when plugging your own product.

Edit:

The Conjur marketeer posted with an updated text disclosing his affiliation, then eventually removed the post. I've attached a screenshot here for posterity since it is relevant to the following discussion. Also to be clear on the above, when I say I "deleted" his earlier comment, I simply removed the text from the original comment and left it blank; only the poster can actually delete it.

[jefferai]

The next major version of Vault will also be supporting versioned secrets.

[bkrodgers]

John -- good of Jeff to allow you to do it as long as you disclose, but it's still tacky as hell to be trolling the issues list of competitors trying to push people to your product.

[ekristen]

Agreed, tacky to be posting.

[c4milo]

I don’t think this is the place to market other tools or even Vault itself, it is out of place and disrespectful. If I wanted to find an alternative tool I would just google for it. I didn’t sign up to Cyberark’s marketing mailing list so please keep the spam out of the technical discussion. Thanks.

[jbscare]

The next major version of Vault will also be supporting versioned secrets.

Ooh, cool. Just to double-check, more like 0.10, which one might call a minor version, as opposed to 1.0? :^)

[jefferai]

@c4milo At a macro level, it's not really different from when others who have built open source tools that can address areas where Vault is lacking helpfully clue in users to their existence. It's a fine line, though. The "tacky" comments likely stem from the fact that Conjur is a commercial competitor to Vault, both of which have paid versions; it is reasonable to look at the post less as an attempt to be helpful and more at an attempt to gain market share, especially given the tone of it and unusual amount of detail/best practices present. But, within reasonable tolerances I'm fine with a plug for Conjur to solve a distinct problem area just as I would be with any other open source product.

@jbscare 0.10 :-) We teased it a bit at https://twitter.com/mitchellh/status/937781924076052480 but generally we don't discuss roadmap very far in advance so I haven't updated this ticket before.

[Tensho]

GitHub announced reports for the such situations.

[mp911de]

The implementation in version 0.10.0-rc1 of versioned secrets breaks all existing Vault clients that are not Vault CLI.

Clients are forced to prefix all secrets with data/ and nest payload within a {data:{…}} object. Especially the latter requires clients to be aware of versioning especially when writing data. Also, metadata about versioning is returned within {data:{ data:{…}, metadata:{…}, …}} instead of the top level object {data:{…}, metadata:{…}}.

[jefferai]

The implementation in version 0.10.0-rc1 of versioned secrets breaks all existing Vault clients that are not Vault CLI.

They continue to work normally. What you're talking about is the default secret/ mount. On new installations it is versioned, and this does require API changes. However, it is easy to replace, existing mounts are not touched, and generally we don't see that actually being used in most installations (people replace with their own mounts according to their data layout needs.

Also, metadata about versioning is returned within {data:{ data:{…}, metadata:{…}, …}} instead of the top level object {data:{…}, metadata:{…}}

There is no top-level metadata object. The metadata being returned here is scoped to the backend data. Any future top-level metadata object would be metadata about the request or response itself.

[jefferai]

In Vault 0.10!