root keyring replication across regions

[benvanstaveren]

Proposal

As it stands at the moment, to use the new Workload Identity integration with federated clusters, one cannot just use the example given in the documentation due to the jwks endpoint on a Nomad cluster being for a single cluster only. The only way so far seems to be: export the jwks keys for each cluster and import them into the vault jwks auth config. This, of course, is not ideal since it's a manual operation and coming from something that "just works" that's a regression.

Ideally the keyring is replicated from the primary cluster (authoritative_region) to all federated members, this seems to be blocking on an open issue but as @tgross mentioned in a comment on #20097 once #14852 is resolved, it could be a possibility.

Personally I'm in favor of this proposal, anything else seems (to me, at least) to require either external tooling, or changes to Vault. The former being, again, a regression in ease-of-management, the latter perhaps not being such a hot idea because it's not that great for separation of concerns.

Use-cases

Makes the migration to workload identity based vault authentication a heck of a lot easier because "things just work" (which is the current situation), and there is no regression and potential additional points of failure brought on by human inattention 😅

[lgfa29]

Thanks for the suggestion @benvanstaveren!

I have placed this into our backlog for further triaging and roadmapping.

[benvanstaveren]

I'm going to bump this because the word is legacy vault integration will be removed, supposedly, in 1.10 and to date, there is still no info/docs on how to make this work with federated clusters where I'm not having to deal with an external bit of tooling to synchronise the jwks keys.

I'm honestly a little disappointed in the vagueness that's surrounded this "feature" since it's inception, and the seeming idea that adding complexity to an area where failure means failing hard and getting 3am phone calls to please fix things is an okay thing to do. Not to mention that the legacy integration removal release keeps getting shoved forward which makes planning for upgrades an absolute pain in the ass.

[tgross]

@benvanstaveren replicating the keyring isn't currently on the near-term roadmap. It seems like you have an implementation issue with Vault but to be honest you've complained about a workaround for a problem you haven't fully described.

My expectation for JWKS and Vault is that if you're sharing a Vault cluster between Nomad regions that each Nomad region would have its own auth config in Vault. In which case there's no need to replicate the keyring as far as I know?

[benvanstaveren]

It seems like you have an implementation issue with Vault but to be honest you've complained about a workaround for a problem you haven't fully described.

Wow. Little passive/aggressive there maybe? I mean, we can go there if you'd like, but I think I've been nice and polite enough... anyway, moving on.

My expectation for JWKS and Vault is that if you're sharing a Vault cluster between Nomad regions that each Nomad region would have its own auth config in Vault. In which case there's no need to replicate the keyring as far as I know?

My expectation for what will be a very breaking change is that it doesn't require a raft of changes to be made as a workaround that will more or less double our spend for a month or two. And a workaround isn't a workaround if it becomes a permanent fixture.

Let me, again, explain the problem. I have 4 clusters. In these 4 clusters I have many jobs that use vault secret mounts to read KV values, get dynamic DB credentials, and so forth. For these things, I have policies in place that guarantee least-access.

Per the documents. I need to define these roles (to include said policies) on the JWT auth endpoint. Now; for KV access this works (even though I'd need to spend an awful lot of time moving things around in the KV store so I can use a templated policy, as shown in the docs). Except I'll need to define 4 policies in Vault (because the mount ID is part of the template), and since roles on the JWT auth endpoint are not shared between other JWT mounts, it also means I need to define a role multiple times if I want to be able to easily move a job from one cluster to another.

Given my use case, I need to be able to run any job on any cluster. And frankly given the documentation? Doesn't seem possible without a lot of duplication and external tooling. This makes it more fault prone than the current way of doing it.

This is what I understand from the documentation. If I'm wrong, I'd love to hear how it'd be done right. Heck I'd love to see a guide or tutorial in the docs for this one (federated clusters, etc.)

Sure, you can work around it by consolidating 4 clusters into 1. Sure, you can work around it by having external tooling. But in my use case, the former is most definitely not an option, and the latter is a regression compared to how things are with the "legacy" method.

Thus; keyring replication because I want every cluster to use the same auth mount. Because that'd be the easiest way; point Vault at the authoritative regions' Nomad servers, and presto majesto, all is well, with only a few small changes to make.

Maybe I'm misunderstanding the implementation details, but again, this is what I have understood from the documentation currently available.

[tgross]

Ok, thanks for a little more detail. Your understanding of the current state seems right to me. Each Nomad cluster needs its own auth config and yes, you'll need separate roles/policies because each auth config is its own Vault mount. I think the assumption from our prior user discussions was that users want regions to be distinct failure domains much of the time and would actually have separate Vaults for each region. Or even if they do, that this wouldn't be terribly burdensome using Terraform (or the like) to drive the Vault configuration. The docs could use some improvement here for sure.

Fortunately at this point with the changes we made to the keyring for 1.9 (KMS and keyring-in-Raft), it's at least feasible to replicate the keyring between regions without it being a risky change. Each cluster would need the same KMS configuration, of course, and the KMS itself would be a single point of failure between regions.

[tgross]

Internal cross-reference: https://hashicorp.atlassian.net/browse/NET-10240

[benvanstaveren]

... want regions to be distinct failure domains much of the time and would actually have separate Vaults for each region. Or even if they do, that this wouldn't be terribly burdensome using Terraform (or the like) to drive the Vault configuration.

Yeah we're kind of weird in that aspect but we have 4 separate clusters due to a variety of legacy reasons and some contractual obligations, it was never meant to develop in that way but it did - for Vault we just have the one instance because we do occasionally need to move the same job to a different cluster altogether and it was just easier to have 1 HA Vault instance than it was to somehow synchronise secrets between different vaults. (Not that it's impossible, just that we'd need an awful lot of automation that needs to be absolutely bulletproof).

Vault configuration/terraform wise, we don't use Terraform at all (I know, boo, hiss), and while it's certainly also doable with Ansible, I am basically a team of 1 in a company of 15, so the more automation and "it just works" behaviour can be had the better it is.

For our use case at least, if Nomad's JWKS keyring can be replicated, that'd be okay for us - perhaps hide that behind a configuration setting somewhere (so the option exists to have the current behaviour at least); as far as it being a SPOF, that's a risk we can live with :) Perhaps something for the vault team to allow the JWT auth endpoint to cache and use "stale" keys if it can't fetch them from the URL or something.

Anyhow, sorry if I was a bit snippy, it's been one of those days :)