Feature request: Google Cloud Platform backend

[devth]

Using Google's (Alpha) IAM: https://cloud.google.com/sdk/gcloud/reference/alpha/iam/service-accounts/

[BarryBotha]

https://cloud.google.com/iam/docs/overview

[skyzyx]

:+1:

[kri5]

:+1:

[DanailMinchev]

@jefferai @pearkes

Are you considering adding Google Cloud support? It will be nice to know what are your plans?

Thank you!

[mzuo]

+1, anyone knows if there is roadmap to add Google Cloud Platform support?

thanks!

[awellock]

+1 It would be awesome to here if there is a plan for this!

[gtaylor]

Please don't leave +1 comments. It spams everyone subscribed to this issue. Hit the emoji on the issue itself.

[fearphage]

Hit the emoji on the issue itself.

Also there's a subscribe button in the right column if you want to stay up to date on new developments.

[mayakacz]

Hi all, If you're asking about an auth backend, please see: https://github.com/hashicorp/vault/issues/2813 If you're asking about a secret backend, stick with this issue. Thanks!

[alexkreidler]

Hello everyone! I am taking a stab at this.

I am basically porting over the AWS secret backend to use the GCP IAM API, because the backend already has a lot of what we need implemented. Currently I am creating Service Accounts with keys instead of AWS users, am planning on dropping STS, and will allow the user to specify a predefined role or upload a new one (although custom roles are still in beta). I know the GCP limit for Service accounts is much lower than the AWS limit for Users (100 vs 5000), but I don't think that should be an issue for most users.

Let me know if there's anything I'm missing. I'll link to a WIP PR when I get it up.

[mayakacz]

Hi Tim!

Re: service accounts, makes sense. You're probably going to want to use user-managed service account keys which can be rotated via the IAM service account API. Google has also created an open-source tool to do this, KeyRotator - might give you some ideas on implementation.

Re: roles, I think that sounds reasonable to allow both pre-defined and custom roles.

If you have a design doc in the works, would love to see it. Do you have a rough timeline (are you doing this for something particular) or is this just something you're kicking around?

Thanks!

[emilymye]

@Tim15 Just some thoughts from when I was thinking about this earlier - are you keeping roles at a project-level only? Just in case you aren't aware, the setIamPolicy calls are to assign policies per resource. That means serviceAccount.setIamPolicy would set the policy on that service account, and you would need to use Cloud Resource Manager's projects.setIamPolicy to set project-level roles. You can see how this quickly balloons in complexity once we want to support anything other than a project/organization-level policy.

[alexkreidler]

@mayakacz User-managed service accounts keys sound good. I don't have a specific timeline but I hope to finish this in the next week or two, depending on use-cases and details.

@emilymye I am quite new to GCP IAM, so forgive me if I have misunderstood.

I think there is a difference between role and policy. Role is a list of permissions, a policy is a map from roles to users. Policies are hierarchical, but roles can only be defined at the project or organization level, and there are built-in global roles as well.

Unlike the AWS IAM system, there are no "inline" roles, so if we want a user-defined role, we have to assign it at either the project or organization level. I am inclined towards the project level.

As far as policies go, I was initially inclined to set the policy at the serviceAccount level using serviceAccount.setIamPolicy as you mentioned, solely based off the principle of least privilege. However, I have a question that I couldn't figure out from the docs. Does the scope of the policy affect the scope of the role? e.g. If I set a policy with the role appengine.Admin on a serviceAccount, will the service account be able to access all of the appengine resources in the project, or even the organization? I am pretty sure the answer is yes, but I want to be sure.

I think this is better than alternatives because:

• When we delete the serviceAccount after the lease is up, the policy goes away too. IDK what to do about user-specified roles, I think in the AWS backend they are "inline", so they disappear along with the user. Since there's a limit of 300 custom roles per project/org we'll probably delete roles as well, but that would stop them from being used in additional requests
• It requires lower-level permissions from the end-user. It only requires iam.serviceAccounts.setIamPolicy (which the user likely already has) vs resourcemanager.projects.setIamPolicy or <topic>.<resource>.setIamPolicy.
• It has a certain amount of locality and is theoretically "owned" by the service account and by extension Vault, so users won't worry about manually editing it. This contrasts with a project level policy that unwary users could stumble upon and edit, or add additional users to, who then get deleted when the lease is up, etc, etc.

The alternative is to have one or a few policies at the project level. I'd like to hear if anyone has use-cases that would take advantage of this.

You can see how this quickly balloons in complexity once we want to support anything other than a project/organization-level policy

I agree that each google cloud api/service providing their own setIamPolicy endpoint is kind of ridiculous. We would have to import and write code for each service. However, I did see: POST https://cloudresourcemanager.googleapis.com/v1/projects/{resource}:setIamPolicy

IDK if this would allow us to set the policy for any {resource}? The docs aren't clear: REQUIRED: The resource for which the policy is being specified. See the operation documentation for the appropriate value for this field

I'd love to hear any feedback on these ideas!

[alexkreidler]

I see the GCP auth backend is implemented as a plugin. Is there a specific reason for that? I see both the AWS secret & auth backends are built-in (and share a utility/helper file).

[emilymye]

@Tim15 I think I maybe wasn't clear, especially since a lot of the IAM is not super intuitive and took me a while to figure out.

1) Many roles can be assigned at other resource level. If you look at the list of roles here, some of the resource types are "project" but some are more specific.

2) I think your understanding (and mine initially, so it is understandable) was that serviceAccount.setIamPolicy grants the {resource} service account permissions, but it actually grants other IAM entities (users, service accounts, or groups) permissions ON that service account.

For example, serviceAccount.setIamPolicy will set the policy for roles acting on that service account resource, i.e. roles like roles/iam.serviceAccountUser or roles/iam.serviceAccountKeyAdmin that specifically act on a service account.

Thus, to actually assign a service account (let's call this serviceAccountX) some role Y on a list of resources, we need to, for each resource:

1) Call getIamPolicy on that resource using that resource's specific API (i.e. CloudResourceManager for project, Compute for compute VMs) 2) Add a "role Y: serviceAccountX" binding to that policy. 3) Call setIamPolicy using the same API/resource.

The Vault team would like new backends to be contributed as plugins, and the AWS backend was written before the plugins existed (calvn wrote the plugin backend code for hashicorp if you are interested). I'm about to add helpers to Vault main and we can look into moving some utils into the main Vault repo/a seperate repo.

[emilymye]

@Tim15 sorry to bug - are you still working on the secrets backend?

[tallpauley]

@emilymye Do you have any plans to implement this, since @Tim15 seems to not be working on it anymore?

[emilymye]

@tallpauley yep! I'm planning to get a design doc posted here in the next few weeks for discussion/feedback

[emilymye]

Sorry for the delay!

Proposal: https://docs.google.com/document/d/1KEpq2ijgck2hRbYbkHzd7DrPtmUG3LyJ3pRWYdPfsJ4/edit?usp=sharing

[tallpauley]

@emilymye The proposal LGTM on first read. My only concern is that we won't be able to utilize this plug-in for gcloud CLI. I get your points about not generating keys: it's slow, and they don't have any TTL built-in like tokens. I also get that this backend proposal is sufficient for using curl or other Google client APIs, but I don't see how we can use it for gcloud.

Obviously it is up to gcloud team to provide a way for us to authenticate w/ the OAuth token directly, but just curious if you have any insights on this. The only way I know to give gcloud a token directly is to modify the sqlite DB it maintains at ~/.config/gcloud/access_tokens.db, which would be super hacky.

My impression is that gcloud is pretty opinionated on auth, it seems to be set on authenticating w/ a key and it managing the tokens. I would submit an issue on gcloud repo, but I can't even find it on github. Is it public?

Hope I'm not going way too far out of scope here, I'd just be really bummed if I can't use your new backend w/ all our gcloud deploy code :) Maybe there is a way we can get the gcloud team involved.

[tallpauley]

Terraform is another interface that relies on GCP keys, not tokens, so these users I'm guessing will have similar concerns (though since TF has public repo, it'd probably be easier to contribute and make direct token auth possible).

[tcolgate]

It would be a shame if issuing/revoking of keyfiles was not directly support:

• I'm not aware of most of the client libraries supporting injection of a token from an environment variable or file. I'm not sure how, for example, I would use the Go Gcloud SDKs, or kube client with a token. This is trivial with setting GOOGLE_APPLICATION_CREDENTIALS. I'd have to set up client with a TokenSource that make explicit vault calls. This will require dev work for every target language, unless the various Google SDK client libs commit to supporting VAULT_ env vars for token configuration?
• This will require considerably more effort from applications. The refresh wont be able to be handed off to a sidecar container unless the application can explicitly read its token in from a file and watch that file.

At present I have the AWS IAM flow working very nicely, with vault issuing and revoking credentials, I was very much hoping the GCP support would match that. Perhaps issueing of keyfiles could be supported as a secondary (and less encouraged) mechanism?

[tcolgate]

It's probably worth mentioing that we designed out current workflow partly around avoiding having to make devs directly integrate vault token retrieval and refresh. We can possibly revisit that, but it's going to be very painful. I'd be more inclined to fork whatever plugins comes out of this effort and adapt it to issue keyfiles. Retrofitting vault refresh to 50 or 60 microservices in 5 languages is probably more effort.

[emilymye]

@tallpauley @tcolgate These are excellent examples of explicitly needing service account keys that I wasn't aware of. We mostly were considering access tokens since we weren't sure which things actually needed keys, but we can revisit now :)

I've mostly been talking to the IAM team, who have issues with how service account keys are managed and were very strongly against exposing the service account keys if possible, but upon closer inspection, it seems like other Google tooling support just isn't supporting access tokens right now. I'll confirm but it looks like all the gRPC libraries just don't allow for access_token.

As a follow-up then, I think one of my main issues is service account creation time. Would the following make sense?

• Still have an explicit service account created at role-set creation, and have two types of role-sets: one for generating access tokens only and one for generating service account keys. That way, you can still have some seperation in case the project owner really only want to give short-lived access tokens. Service account keys for the second type will be generated on the fly and managed by Vault.
• Downsides: Each role set has a hard limit of 10 secrets per role-set (explicit 10 for now). If you really need more secrets per role-set, you can duplicate the role-set with a different name? Not sure if this is the best approach (vs new service account per secret, multiple service accounts per role set). I really don't want to create service accounts on the fly.

Sidenote: @tallpauley I don't think gcloud is open source and it is very confusing (most people seem to file on https://github.com/GoogleCloudPlatform/google-cloud-common out of confusion, but I think it's actually supposed to be filed as a product issue? In any case, it looks like they mostly got lost, but there are a couple of issues floating around various google cloud libraries (https://github.com/GoogleCloudPlatform/google-cloud-common/issues/76 and https://github.com/googleapis/nodejs-common/issues/11).

@mayakacz fyi

[tcolgate]

@emilymye For 90% of our use case, the role-limit wouldn't matter too much, since most things hitting gcloud services will have < 10 instances. For those that need more we could stick with our existing process of storing a gcloud keyfile in vault directly. Unfortunately the limit is just low enough that we'll probably have to bake a warning.error into our internal release tooling.

If the google SDKs supported configuration of a token source via environment variables, that would also work (in that I could have our sidecar act as a token source), and that would remove the need for Google to have vault specific logic in the clients. I suspect that's quite a tough call though.

I think the equivalent of what AWS does would actually be to create a service account and issue a key, with lease expiry removing both. What's the primary objection to creating service accounts?

[emilymye]

@tcolgate GCP has a fairly low default limit on service accounts (100/project) and service account creation/permission propagation can take a long time (up to 1 min). If we create the service account when we create the role-set (which is more of an admin/set-up step), it means:

• you wouldn't have to wait per-secret for a service account
• we can generate more service account keys (~100 (role-sets) * 10 (keys/role-set) or infinite access tokens vs just 100 keys overall)
• If you hit the limit of service accounts/role-sets, you can request a quota increase, which would be better at a set-up step rather then at 'run-time' (secret-gen)

At this point, I'm pretty inclined to go with having both access token and service accounts key generation possible (with the 10 key/role-set limit) and documenting very carefully.

[tcolgate]

@emilymye can you just clarify, can the 10 keys/role-set limit be up'd by requesting a quota increase, or is that hard limit? Does deleting/revoking a key immediately free up the quota? (e.g. if I am deploying an app with 8 pods, deleting 1 pod, freeing the key, and starting another pod and requesting it, can I expect to stay in quota and rolls over new pods in a reasonable time frame).

[emilymye]

@tcolgate I believe the 10 keys/role-set is a hard limit set by the IAM team, who would probably lower the limit if they could - they really want to minimize how many long-lasting private keys per account are out in the wild. Key deletion is instant so as soon as Vault deletes a key the service account should be able to add another.

[tallpauley]

Sorry, late to weigh in on your response @emilymye. Thanks for the link to filing a product issue for gcloud, that is useful.

@emilymye I'm sorry you have to complicate your design to compensate for Google's lack of coordination on the IAM front. You're totally making the best of what you've been given, I just really wish Google would either:

1) Unify authentication of client libraries and gcloud around tokens, since IAM team seems to prefer generating a bunch of these as they have TTLs. 2) Allow service keys to have TTLs, speed up generation of them, and take off the seemingly arbitrary limits.

I don't care if it's tokens or service accounts, Google just needs to take the cue from Vault/AWS combo and provide a way to generate temporary access to talk to Google APIs via Vault, in a format that is supported across Google tools and libraries.

Making multiple role-sets to get around limits sounds painful and unnecessarily complicated (obviously not your fault @emilymye). I don't think the vault backend should try to protect us from Google's limits, I'd rather just get the quota error and we talk to Google if I need more.

Sorry if this is unhelpful @emilymye, I just wish the IAM team (or GCP as a whole) could read this and give you and us the experience that AWS customers are getting w/ dynamic service key creation via Vault.

[emilymye]

@tallpauley np! It's unfortunate that the authentication story seems to be varied across APIs/libraries/gcloud but things are always changing so we'll just have to see. I'd also say it's probably pretty hard to convince some users to deal with things that have short TTL vs long-lasting service account keys, even if it's not the most secure so ¯\_(ツ)_/¯

Just as an FYI for anyone watching the thread, just wanted to quickly update - we're actually almost done so please stay tuned! As a summary, we're:

• Allowing both token and key generation (rolesets can generate one secret type)
• Tokens will be non-renewable, 1hr max TTL
• Key TTL/MaxTTL are based on system defaults or configured in secrets engine. Each role set has a max 10 keys and will just return an error from GCP if you hit the limit. Ways to get around this: use tokens if possible, release unneeded keys more often, create more role sets.

Thanks, everyone!

[emilymye]

The secrets backend is in Vault as of v 0.10.0. Please try it out and file issues/FR at hashicorp/vault-plugin-secrets-gcp!

@jefferai we can probably go ahead and close this issue? I'm don't think another backend was proposed here.

[jefferai]

Yup!