search

hashicorp / vault

A tool for secrets management, encryption as a service, and privileged access management
https://www.vaultproject.io/
Other
30.91k stars 4.18k forks source link

Proposal: JWT Claim-based OIDC Auth Backend #2525

Closed mwitkow closed 6 years ago

mwitkow commented 7 years ago

Kubernetes supports authentication (and group extraction forth authorization) using OICD (OpenID Connect) JWT id_tokens tokens, see here for docs. Basically JWT tokens are crypto-verifiable JSON key-value pairs called "claims".

For Kubernetes Auth, two such claims are used:

Both KeyCloak and Dex are configurable OpenID Connect servers that can delegate to upstream identity providers (e.g. Azure or Google).

This proposal is about introducing an Auth Backend that is a configurable, generic OICD backend that uses JWT token validation.

Contrary to what's been discussed previously in #465, OIDC doesn't require browser flows to be used, and such is not an obstacle for Vault adoption. They can be used in exactly the same fashion as GitHub personal tokens, by copy-pasting.

 vault auth -method=oidc token=<id_token>

In fact this is exactly what K8S's kubectl is expected to be used, with --token flag.

A couple of other considerations:

The K8S oicd plugin seems fairly straightforward and could act as a basis for this work. We'd actually be willing to send in PRs for this if Vault maintainers would accept them :)

jefferai commented 7 years ago

No need to fork at all. I have some ideas. Like I said, just give me some time to a) think on it and b) discuss with the team. But also trying to make sure I have all the context e.g. Eric's most recent comment.

ericchiang commented 7 years ago

@jefferai If you accept an access token to talk to the userinfo endpoint, you can't determine what client issued that access token. So as an attacker I could create an arbitrary client with Google, login to my own client, then send that access token to Vault and Vault will log me in.

Sorry, "won't work with Google" is maybe a bit strong. What I mean is anyone will be able to talk to Vault, and it's up to Vault to do the ACLs.

If you only accept ID tokens, you can ensure they went through a trusted OAuth2 client, and not any random client that Google issued.

Edit: that's what I was trying to point out in this comment https://github.com/hashicorp/vault/issues/2525#issuecomment-292586980

jefferai commented 7 years ago

If you accept an access token to talk to the userinfo endpoint, you can't determine what client issued that access token. So as an attacker I could create an arbitrary client with Google, login to my own client, then send that access token to Vault and Vault will log me in.

Only if the claims from the userinfo endpoint match the claims configured on the role, and the info from the userinfo endpoint is dependent on your token.

This isn't to say that there aren't spoofing concerns on both sides, especially in cases where you're looking for user-defined information. e.g. if a user can set an arbitrary claim in their account that Google will sign when issuing an ID token, and that's all you're matching on, you are opening up yourself to spoofing; same if you only match that from the userinfo endpoint.

ericchiang commented 7 years ago

Only if the claims from the userinfo endpoint match the claims configured on the role, and the info from the userinfo endpoint is dependent on your token.

Correct. If Vault enforces the ACLs correctly then it should be okay. We've shied away from doing this in Kubernetes, but I'm not as familiar with the Vault's auth-Z layer. Just felt it was important to call out.

bkrodgers commented 7 years ago

With an access token, if that token is generated out of band (including directly from the CLI calling the IDP), Vault on the backend would need to validate the access token. That would give it the opportunity to see the client ID that created it and ensure it is what it expected. The problem is that validating reference access tokens is not currently in a finalized standard. There is one in draft, but right now each provider has its own custom grant extensions for that. Some providers also support JWT access tokens, which can be validated without calling the IDP, but that's not universal either.

By having the back end make the call to get the token, it received the token directly from the IDP and can thus rely on TLS for integrity and skip that validation. But otherwise Vault would need to validate the access token and its client ID.

jefferai commented 7 years ago

@ericchiang Sure, appreciate you making sure we're aware!

@bkrodgers A userinfo endpoint would validate the token implicitly. If the administrator configures the correct endpoint (and it's over TLS, of course) then if the endpoint returns correct data with correct claims for the token provided by the client it should be appropriately validated.

Just as a heads-up, we're also having some internal discussions about this and we'll get back to you all with some more concrete thoughts after we're done. (These discussions are not only "how should we do X", but like, if it's done way X or Y, what's the overall maintainability story, what are we wanting to take on, what lessons have we learned from prior work, etc.)

It's late on the east coast today and next week we have several teammates out on vacation and/or trips so we'll be a bit slower than usual. But we'll definitely keep an eye on this.

bkrodgers commented 7 years ago

Cool, thanks. I appreciate the thought and input from everyone. Hopefully we'll come to a solution that will meet everyone's needs.

Userinfo endpoint would be validating the token, though it won't tell you what client ID created it, if you care to validate that. To check the client it'd either need to be a JWT access token or send it to the IDPs token introspection endpoint. Here's the draft standard I'm referring to for that: https://tools.ietf.org/html/rfc7662. Not sure if anyone's started implementing it yet.

mikeokner commented 7 years ago

2525 (comment) for example.

Sorry for the confusion. I misread that as the Vault server implementing the flow, not the CLI specifically. In my PR, the CLI just collects the credentials from the user and passes them to the server. The server performs all interactions with the oauth provider.

Personally, I don't think it would be a serious hurdle to handle several possible configurations within one backend that would hopefully make everyone happy:

Which flows a particular Vault instance would support would just depend entirely on how an admin decides configure the backend and their provider.

jefferai commented 7 years ago

Userinfo endpoint would be validating the token, though it won't tell you what client ID created it, if you care to validate that. To check the client it'd either need to be a JWT access token or send it to the IDPs token introspection endpoint. Here's the draft standard I'm referring to for that: https://tools.ietf.org/html/rfc7662. Not sure if anyone's started implementing it yet.

In my mind this is outside of Vault's scope; the identity server introspects when you call the userinfo endpoint so it's really up to it to decide if the token came from a valid client. Of course, nothing stops you from having the identity server attach claims that indicate the client and use that for matching...

@mikeokner It's trickier than you might suspect given Vault's architecture and the way it isolates backends from the core, but I've got some plans coalescing in my mind and in discussion with the team. It might be mid-late next week before there's a concrete proposal, but something will come -- I promise!

ericchiang commented 7 years ago

Just realized another issue with using the access token I'd like to run past everyone.

If I configure Vault with Google, and declare that "(my account)@gmail.com" has access to certain secrets, then I can never login to a non-trusted OAuth2 client using that account, because that client would be given an access token it could use to talk to Vault on my behalf.

Any website that implements a "Login with Google" flow (e.g. nytimes.com) could use the access token it gets from Google to read secrets from Vault. I'd essentially have to have a dedicated Google account just for Vault.

I really think this proposal should consider having an option where in only accepts ID tokens with a specific audience claim, and not access tokens.

On Fri, Apr 7, 2017 at 4:13 PM, Jeff Mitchell notifications@github.com wrote:

Userinfo endpoint would be validating the token, though it won't tell you what client ID created it, if you care to validate that. To check the client it'd either need to be a JWT access token or send it to the IDPs token introspection endpoint. Here's the draft standard I'm referring to for that: https://tools.ietf.org/html/rfc7662. Not sure if anyone's started implementing it yet.

In my mind this is outside of Vault's scope; the identity server introspects when you call the userinfo endpoint so it's really up to it to decide if the token came from a valid client. Of course, nothing stops you from having the identity server attach claims that indicate the client and use that for matching...

@mikeokner https://github.com/mikeokner It's trickier than you might suspect given Vault's architecture and the way it isolates backends from the core, but I've got some plans coalescing in my mind and in discussion with the team. It might be mid-late next week before there's a concrete proposal, but something will come -- I promise!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hashicorp/vault/issues/2525#issuecomment-292673127, or mute the thread https://github.com/notifications/unsubscribe-auth/ACO_XcnSg22yRayBanVrip3QpkIEvk_kks5rtsMGgaJpZM4Ml9kn .

mwitkow commented 7 years ago

Interesting Eric. I kinda start thinking that it is better to separate OIDC and OAuth2, with the former being the recommended one.

On Sun, 9 Apr 2017, 09:04 Eric Chiang, notifications@github.com wrote:

Just realized another issue with using the access token I'd like to run past everyone.

If I configure Vault with Google, and declare that "(my account)@gmail.com " has access to certain secrets, then I can never login to a non-trusted OAuth2 client using that account, because that client would be given an access token it could use to talk to Vault on my behalf.

Any website that implements a "Login with Google" flow (e.g. nytimes.com) could use the access token it gets from Google to read secrets from Vault. I'd essentially have to have a dedicated Google account just for Vault.

I really think this proposal should consider having an option where in only accepts ID tokens with a specific audience claim, and not access tokens.

On Fri, Apr 7, 2017 at 4:13 PM, Jeff Mitchell notifications@github.com wrote:

Userinfo endpoint would be validating the token, though it won't tell you what client ID created it, if you care to validate that. To check the client it'd either need to be a JWT access token or send it to the IDPs token introspection endpoint. Here's the draft standard I'm referring to for that: https://tools.ietf.org/html/rfc7662. Not sure if anyone's started implementing it yet.

In my mind this is outside of Vault's scope; the identity server introspects when you call the userinfo endpoint so it's really up to it to decide if the token came from a valid client. Of course, nothing stops you from having the identity server attach claims that indicate the client and use that for matching...

@mikeokner https://github.com/mikeokner It's trickier than you might suspect given Vault's architecture and the way it isolates backends from the core, but I've got some plans coalescing in my mind and in discussion with the team. It might be mid-late next week before there's a concrete proposal, but something will come -- I promise!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hashicorp/vault/issues/2525#issuecomment-292673127, or mute the thread < https://github.com/notifications/unsubscribe-auth/ACO_XcnSg22yRayBanVrip3QpkIEvk_kks5rtsMGgaJpZM4Ml9kn

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hashicorp/vault/issues/2525#issuecomment-292771070, or mute the thread https://github.com/notifications/unsubscribe-auth/AJNWo6pUGvjBS-E-lSB6IjkbxZcJKNwxks5ruJD2gaJpZM4Ml9kn .

jefferai commented 7 years ago

I really think this proposal should consider having an option where in only accepts ID tokens with a specific audience claim, and not access tokens.

So you submit an oauth2 token, hit up the userinfo endpoint, and it returns claims including aud. You match the aud claim. What's the problem exactly?

ericchiang commented 7 years ago

Userinfo isn't required to include aud claim.

http://openid.net/specs/openid-connect-core-1_0.html#UserInfoResponse

On Apr 9, 2017 5:18 AM, "Jeff Mitchell" notifications@github.com wrote:

I really think this proposal should consider having an option where in only accepts ID tokens with a specific audience claim, and not access tokens.

So you submit an oauth2 token, hit up the userinfo endpoint, and it returns claims including aud. You match the aud claim. What's the problem exactly?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hashicorp/vault/issues/2525#issuecomment-292782433, or mute the thread https://github.com/notifications/unsubscribe-auth/ACO_XdJCcytS1mydbPtiEcByfJNHUO49ks5ruMyQgaJpZM4Ml9kn .

jefferai commented 7 years ago

It may, and if the response is signed, it (according to spec) SHOULD. And either way if you are requiring an aud match and it's not part of the userinfo, just reject because there's no match. Easy-peasy.

bkrodgers commented 7 years ago

At least with our current setup with Ping, the UserInfo endpoint doesn't send that data back signed, and thus doesn't include the aud claim. I'd need to look further into whether it can be configured to include that. However, a JWT access token will have client_id in it, or call the token introspection endpoint for a reference token. The issue with the latter is that it isn't a finalized standard yet. However, Ping has already implemented it, and perhaps others have too.

jefferai commented 7 years ago

@bkrodgers I thought Ping didn't give you JWT token after OAuth is complete? If so, what's in it?

bkrodgers commented 7 years ago

Ping won't give me a ID Token in resource owner password flow, but it can be configured to give me an access token in JWT format. We can configure what we want to be in it, but here's an example:

{
  "scope": [
    "openid",
    "profile",
    "email"
  ],
  "client_id": "<client-id>",
  "iss": "<our-ping-server>/as/token.oauth2",
  "sub": "bkrodgers",
  "user_id": "bkrodgers",
  "OrgName": "Monsanto",
  "group": [
    "group1",
    "group2"
  ],
  "exp": 1491857773
}

So for such a token, Vault could check the client ID by validating the JWT and looking at the client_id field. To make it work with reference tokens and still validate client ID, Vault would need to call out to the token introspection endpoint.

jefferai commented 7 years ago

Wait, so Ping refuses to give a standard OIDC token in that flow but will give a signed JWT containing groups and a client id and a user id?

bkrodgers commented 7 years ago

If you tell it to, yes. They support either JWTs or reference tokens for access tokens, and you can configure what data you want included. But I haven't found a way to get it to tell you to give it an ID token. I did put a question into Ping support to confirm that isn't possible, but I'm pretty sure you can't.

jefferai commented 7 years ago

Mumble mumble mumble PITA mumble mumble

OK, so I'm trying to scope out some work, so here are my next two questions, applicable to everyone:

1) Realistically, how many providers are out there that support only oauth2+userinfo (or some in-between bastardization like Ping that support variants but in a way that can be secure) 2) How many providers are out there that support a) normal oauth2 flows (e.g. redirects), b) return OIDC tokens from these flows, and c) support groups and other mechanisms that make them realistic to use for an auth backend to Vault?

bkrodgers commented 7 years ago

Just to clarify, Ping's in-between bastardization only applies to the resource owner password flow, since it isn't included in the OIDC spec and leaves ambiguity to the implementation as to what to do with the OAuth2 flows not specified by OIDC. It handles OIDC according to the spec for the implicit, auth code, and hybrid flows, those just aren't CLI friendly.

I don't have much hands on experience with other providers out there, but I'll see if I can find any info on how they handle things. I know Gluu is one free option I can download and play around with for free, and I'll check out docs for some others.

jefferai commented 7 years ago

@ericchiang @mwitkow just pinging you specifically in case you haven't been following the recent discussion

ericchiang commented 7 years ago
  1. Realistically, how many providers are out there that support only oauth2+userinfo (or some in-between bastardization like Ping that support variants but in a way that can be secure)

Ping is the first I've seen.

  1. How many providers are out there that support a) normal oauth2 flows (e.g. redirects), b) return OIDC tokens from these flows, and c) support groups and other mechanisms that make them realistic to use for an auth backend to Vault?

Including groups, off the top of my head:

I don''t know if keycloack supports groups.

Notable ones that support OpenID Connect but don't provide groups:

Also, as mentioned above. Dex, which I work on, acts as an OpenID Connect proxy. Users login through upstream providers like GitHub, LDAP, SAML, etc. and dex mints ID tokens. It supports groups.

mwitkow commented 7 years ago
  1. Realistically, how many providers are out there that support only oauth2+userinfo (or some in-between bastardization like Ping that support variants but in a way that can be secure)

Yeah, same here, I haven't seen anyone not support OIDC if they have userinfo, because it is a relatively trivial addition that makes it easy to "tick a box".

  1. How many providers are out there that support a) normal oauth2 flows (e.g. redirects), b) return OIDC tokens from these flows, and c) support groups and other mechanisms that make them realistic to use for an auth backend to Vault?

To that I'll add (with groups support):

bkrodgers commented 7 years ago

Looking at the Okta docs, specifically here, it looks like Okta behaves the same way Ping does in regards to the password flow. It doesn't return an ID token for the password flow either. The key difference is that Okta has a (proprietary) extension to the OIDC spec to allow passing a session token to the OIDC authorization endpoint to do API/CLI friendly authentication, which can be obtained using their (also proprietary) authentication API. Via that method, you do get an ID token, it appears, but that's not 100% following the OIDC specs either. So strictly speaking, Okta would appear to behave the same way as Ping as far as not giving an ID token for password grant type, but provides an alternate way to get around using password flow in the first place.

I can't tell for sure if Okta supports calling the userinfo endpoint with an access token obtained via password flow, but nothing I can see in the docs suggests it doesn't allow that. But if it does, the implementation we've proposed in our PR would likely work as is with Okta as well. That's not to say we'd need to deprecate the Okta backend, only that you probably could.

jefferai commented 7 years ago

Just as an update, we had to push back an internal discussion on this due to travel schedules this week, so we should be discussing early next week. The main thing we need to talk about is whether we want to target a backend that can handle the redirect workflow, enabling it to work with many more providers. From a technical level, the way to do this in Vault could dovetail nicely with a couple of other future enhancements we had planned, so it's sort of a two-pronged question right now -- do we want to take on this work, and if so, how to make sure that the design we use to implement that doesn't constrain future features.

If we decide to go down that path, someone on the Vault team would handle the inner core changes, and work with anyone interested on this ticket to handle the needs on the backend. It does push up the scope of the original backend a bit, but not a huge amount; mostly it would change the interfaces and design a bit, which is why I want to have this discussion before commenting back on the original design doc.

mwitkow commented 7 years ago

@jefferai what's the ETA on a decision?

We actually need an OIDC backend in the next two weeks and are willing to contribute to building it. However, if it takes a while, we probably will just internally fork it and build it as specified in the design doc, as it would become a blocker for our internal developer/operator credentials system.

From our point of view, the redirect flow is superfluous, but if it makes a sense if there's wider adoption. Just to clarify, what do you mean by the redirect flow?

jefferai commented 7 years ago

@mwitkow Next week some time.

I'm confused by what you're proposing. Why would you internally fork it according to the design doc instead of just building it to the design doc in a normal public way with a PR and eventual merge upstream?

jefferai commented 7 years ago

client-side callback URL for an AuthCode flow (where the client is a web browser and sees the OIDC provider's redirect) server-side callback URL for AuthCode flow (where the Vault server implements a callback URL for an identity provider)

Server side; generally outside of client applications the client-side callback isn't useful. Since Vault is a remote server it would need to implement that method.

jefferai commented 7 years ago

We were able to have a meeting a bit earlier than planned, so I have some feedback.

Here's what we want to do. It helps to skim #2200 to understand what's being proposed.

1) Have an oidc backend that, at a high level, is able to look at claims and assign policies based on these claims. 2) Define an interface that is suitable for defining how to get the claims. This can take multiple forms, discussed in a moment. 3) Write support for these different providers as plugins against the interface.

Going into broader detail:

For any given provider, details differ, both from a simple OAuth2 perspective and from an OIDC perspective. For instance, whether groups are defined in a groups claim or a roles claim -- to Vault this is a variant that needs to be handled somehow. It would be nice to avoid hardcoding values for different providers right into the backend, and it would also be nice to not have to have the user be aware of this type of behavior.

One way to handle this is to create interfaces that hide the differences. So if at a base level we care about individual users/groups. but not how they're encoded in a given OIDC, we can imagine submitting an OIDC token to login against a backend role with a specific type. Then the implementation of the interface knows how to parse out the unique user and group values for that given provider.

We can take this a step further with providers that don't return OIDC but do allow a method of getting signed claims and using authorized clients, such as the Ping resource owner flow. A given interface implementation can be configured with the client id/secret, and the login path can allow submitting the resource password. The implementation could then turn the returned OAuth2 token into claims by hitting Ping's UserInfo endpoint, and those claims can be returned to the backend for processing/matching to policies.

Going one step further, we can even take this approach and with some work in Vault's core, we can enable a normal OAuth2 server-side redirect authentication flow; this way users don't need to fetch OIDC tokens on their own, but can be given a resource that will resolve to a valid Vault token once they log in on the web via normal redirect flow means.

So similar to what we're doing in #2200, this can be built as a core of common code, a well-defined interface to handle providers of various types, and then plugins to actually enable those providers.

As far as the change of scope for the original proposal, it's actually not much, more a reorganization than anything else. There still needs to be a way to map users/groups/roles/etc. to policies so those API endpoints are necessary (but! there are actually several backends that have similar needs and really need the code refactored a bit into a library, so reuse is potentially possible); but the token parsing and verification will happen behind an interface satisfied by a trivial plugin (possibly a "generic" plugin for OIDC tokens that don't need anything special).

However, an advantage of this plugin approach is that we don't need all of the various providers implemented at once to launch the backend, and because the interface is internal we can update it if/when needed without affecting the public API.

Overall we believe that this will coalesce a lot of user stories around authenticating via OIDC (or OAuth-using identity servers that provide user info) and make it easy to support a growing list of these providers going forward.

Thoughts appreciated!

mwitkow commented 7 years ago

@jeffarai, as far as I understand #2220 is about providing a plugin system for auth. That's great!

However, I have a few key problems

OAuth Plugin or Generic Plugin

The proposal you outlined was fairly clear until:

We can take this a step further with providers that don't return OIDC but do allow a method of getting signed claims and using authorized clients, such as the Ping resource owner flow. A given interface implementation can be configured with the client id/secret, and the login path can allow submitting the resource password. The implementation could then turn the returned OAuth2 token into claims by hitting Ping's UserInfo endpoint, and those claims can be returned to the backend for processing/matching to policies.

At this point it is unclear to me what is the input to this "GenericBackend". Is it a:

While I can see how "Token" is a defined-enough (you have a plugin that takes it and returns you a set of username, roles and groups out of it), "any challenge" becomes murky. Doesn't this basically boil down to: "I got input from user, and return a Token" of what an Auth Backend should do? At this point, Vault is better off exposing a "Generic Auth Backend" plugin system, and we should leave this discussion about OAuth2 and OIDC to rest until that's ready and both can be implemented inside of that.

The Per-Provider Plugin assumption

At the same time, I want to provide a scenario that would challenge the following assumption:

It would be nice to avoid hardcoding values for different providers right into the backend, and it would also be nice to not have to have the user be aware of this type of behavior. Then the implementation of the interface knows how to parse out the unique user and group values for that given provider.

Let's imagine person Y with a more systems-admin background, without much proficiency in Go. They just bought an Acme Auth provider for their Company X. Acme Auth supports OIDC and says: to configure third-party software to OIDC authenticate with Acme Auth (see linked documentation above) do:

The person Y would know how to configure an OIDC backend with Kubernetes, their config takes in values stated in the documentation, i.e. the ones that are also relevant in OIDC. Now person Y looks at Vault documentation and sees: OIDC Dex, OIDC Google, OIDC Keycloak. But no OIDC Acme. And they don't know Go to write a plugin themselves.

As such I propose to have a Generic OIDC backend with the similar knobs outlined in the proposal. It wouldn't be hard to configure for any savvy, and provide much better adoption for lesser known use cases (e.g. we have our own OIDC provider).

Code sharing

As far as the change of scope for the original proposal, it's actually not much, more a reorganization than anything else. There still needs to be a way to map users/groups/roles/etc. to policies so those API endpoints are necessary (but! there are actually several backends that have similar needs and really need the code refactored a bit into a library, so reuse is potentially possible); but the token parsing and verification will happen behind an interface satisfied by a trivial plugin (possibly a "generic" plugin for OIDC tokens that don't need anything special).

I am not sure what you mean here. Do you mean: a) there will be a generic library (used e.g. in Github or Okta) for storing a mapping from a concept of a "user", "group" and "role" to a policy that we can reuse b) the "OAuth2/OIDC hybrid backend" should have one that's common and consumed by all these "plugins"?

If it's a), that's great. Most of my hacky oidc plugin code I have locally deals with that. The rest boils down to ~40 lines. If it's b) I'm more skeptical, as outlined above.

Also by "interface" and "plugin", did you mean the high-level "Database Backend" discussed in #2200 or a "OAuth2/OIDC hybrid plugin"?

jefferai commented 7 years ago

as far as I understand #2220 is about providing a plugin system for auth. That's great!

Only in the sense of database authentication credentials. It's about adding a way for there to be a common core of a database backend whose actual implementations that connect to the DBs and toggle users there are separated into plugins.

At this point it is unclear to me what is the input to this "GenericBackend"

Plugin, not backend -- just want to be clear here that this is about having a plugin system for this backend. Forget what I said about a generic plugin, I think it muddied the waters too much. My point was only that many OIDC providers use spec-defined or common claim names so there is a potential for a single plugin to handle many cases, but other plugins could easily handle variations.

Let's imagine person Y with a more systems-admin background, without much proficiency in Go. They just bought an Acme Auth provider for their Company X. Acme Auth supports OIDC and says: to configure third-party software to OIDC authenticate with Acme Auth (see linked documentation above) do:

set the discovery url to this
use sub claim for username
use acme_groups claim for groups discovery
The person Y would know how to configure an OIDC backend with Kubernetes, their config takes in values stated in the documentation, i.e. the ones that are also relevant in OIDC. Now person Y looks at Vault documentation and sees: OIDC Dex, OIDC Google, OIDC Keycloak. But no OIDC Acme. And they don't know Go to write a plugin themselves.
As such I propose to have a Generic OIDC backend with the similar knobs outlined in the proposal.

I don't really understand the issue/problem here. If they are using standard values that can be handled by a "generic" style plugin, great. (Maybe part of the config of this plugin is the key of the claims to be used for user/groups, so long as the values are expected, e.g. string and []string.) If the ACME auth provider does something custom, they can create a custom plugin that they can maintain on their own that can be plugged into Vault without needing to wait for us to make a release.

mwitkow commented 7 years ago

I don't really understand the issue/problem here. If they are using standard values that can be handled by a "generic" style plugin, great. (Maybe part of the config of this plugin is the key of the claims to be used for user/groups, so long as the values are expected, e.g. string and []string.) If the ACME auth provider does something custom, they can create a custom plugin that they can maintain on their own that can be plugged into Vault without needing to wait for us to make a release.

This is exactly what the current design proposes: parsing JWS claims and allowing the administrator of a Vault to configure claim names fields. And this is what every OIDC (not OAuth2 + proprietary stuff) provider I know does:

They all either encode scopes, roles or groups inside simple OIDC claims (either as string or []string). The proposal outlines hits all of them.

The whole point of my discussion was the following: If you have an ACME Auth provider, and it does OIDC, you most likely will not need custom code and a plugin to work with it.

jefferai commented 7 years ago

Yes, but how do you get that OIDC token in the first place? In many machine cases it might be given to you, but for user cases you're very often going through a web-based flow; otherwise you may be providing credentials which are used on your behalf.

I did say that the scope of the original proposal was not going to have to change much! All that really needs to be done for your needs is for straight up hey-Vault-here-is-this-OIDC-token-I-already-have workflows, the actual parsing and/or verification will happen inside a simple plugin rather than directly in the backend code. It's a shifting of where this happens, not a shifting of what happens.

But this has a number of benefits down the line -- it makes it easier to write plugins that allow going through the entire flow from the user perspective (e.g. the web redirection flow). It also means that anyone that needs to do something slightly-to-very different from anything an existing plugin can handle only needs to fork and maintain small plugin that can be loaded dynamically rather than fork and maintain the entire Vault codebase. (And, if it's just a small quirk, that can likely be handled by a config flag to an existing plugin rather than a new one.)

So the additional bits to what you originally proposed are mostly a) adding plugin logic, which is mostly going to be from helper libraries or a bit of copypasta from #2200; and b) defining the plugin interface. I have faith that the various interested parties on this ticket have enough experience to help that take shape.

mwitkow commented 7 years ago

actual parsing and/or verification will happen inside a simple plugin rather than

My question above was about the scope of the "plugins":

If it's the second, I agree. If it is the latter, it feels like scope creep entering Generic Auth territory. If it is the second, I assume you're talking about an interface similar to:

type TokenVerifier interface {
   Verify(context.Context, string) (username string, groups []string, roles []string, error)
}

But at this point, wouldn't it just be better to have a "Generic Token" authenticator with a Webhook configuration similar to: https://kubernetes.io/docs/admin/authentication/#webhook-token-authentication?

user cases you're very often going through a web-based flow

Why would you want to implement the web-based flow on the server side? Neither OAuth2+UserInfo, nor OIDC needs it. Doing the Web based auth flows server side is complicated:

In our implementations internally (for our CLI tool), and I presume in Kubernetes (see here, @ericchiang will know better), the OIDC AuthCode browser-based flow is done inside the client with a "local web server" for the duration of the callback. This makes it more secure (the CSRF is unlikely as the endpoint lives for a short period of time) and easier to customize for users, since they only need to "fork" a client and the interface is just the "Token" itself.

loaded dynamically rather than fork and maintain the entire Vault codebase

Is this really where you want to go with Vault? I mean, I thought it would be an extensible piece of software with well defined "composable", but "service-oriented" interfaces. Do you think users would trust a dynamically loadable module system?

The point I'm trying to make here is: I actually really like Vault as an auth system, but I would prefer it to do the least possible thing and make it easy to wrap your head around. Dynamic plugins, interfaces and preloaders are hard to reason about and be able to "judge" their sanity. I'd much rather have few standards-based or generic backends that people can integrate things separately.

jefferai commented 7 years ago

If it is the latter, it feels like scope creep entering Generic Auth territory

What is "Generic Auth territory"?

But at this point, wouldn't it just be better to have a "Generic Token" authenticator with a Webhook configuration similar to: https://kubernetes.io/docs/admin/authentication/#webhook-token-authentication?

A plugin could certainly behave in this way, but it doesn't require them to, e.g. if the submitted login information (say, a fully formed JWT OIDC token) has enough information in it to avoid a call to a third party.

Why would you want to implement the web-based flow on the server side? Neither OAuth2+UserInfo, nor OIDC needs it.

This assumes you already have an OIDC or OAuth2 token. But we've had plenty of requests (and some PRs that we have so far declined to merge because of their lack of genericism) for auth backends that can handle the OAuth2 flow, because they don't want users to have to dig through UIs or APIs to get a token for Vault, they want their users to simply authenticate Vault directly. This lets Vault send a redirect address, have the user click on it to authorize Vault, then have Vault return a token to the user.

In our implementations internally (for our CLI tool), and I presume in Kubernetes (see here, @ericchiang will know better), the OIDC AuthCode browser-based flow is done inside the client with a "local web server" for the duration of the callback.

Vault doesn't have a way to display these pages. If you're doing this via textual means I'd be interested in that mechanism. I know the OAuth2 flows well but not the OIDC flows as much.

Is this really where you want to go with Vault? I mean, I thought it would be an extensible piece of software with well defined "composable", but "service-oriented" interfaces. Do you think users would trust a dynamically loadable module system?

Doesn't a plugin system add extensibility and composability? Why should users not trust plugins that their administrator has authorized?

The point I'm trying to make here is: I actually really like Vault as an auth system, but I would prefer it to do the least possible thing and make it easy to wrap your head around.

The problem is that I need to look at it from the perspective of a thing that solves use cases for many customers. I don't really understand the consternation you have with what I proposed, as for your particular needs the complexity is not significantly greater.

Dynamic plugins, interfaces and preloaders are hard to reason about and be able to "judge" their sanity.

I think this depends very strongly on the specifics: when are plugins loaded, what are the interfaces, etc. I don't know where "preloaders" is coming into the conversation though... :-)

I'd much rather have few standards-based or generic backends that people can integrate things separately.

You want a backend that consumes OIDC tokens. I want that backend, when used with different providers, to be able to consume OIDC tokens generated directly through Vault's actions. This is directly as a result of repeated user requests and needs. I don't think we are at odds here; I just want more functionality to be able to be supported, and am proposing that a plugin interface be used to keep this complexity out of the core backend code to the greatest degree possible.

mikeokner commented 7 years ago

You want a backend that consumes OIDC tokens. I want that backend, when used with different providers, to be able to consume OIDC tokens generated directly through Vault's actions. This is directly as a result of repeated user requests and needs. I don't think we are at odds here; I just want more functionality to be able to be supported, and am proposing that a plugin interface be used to keep this complexity out of the core backend code to the greatest degree possible.

I agree with this. While @bkrodgers and I certainly have the ability to stand up some obnoxious, one-off application that does nothing other than go through the browser flow so some poor user can copy/paste a long, encoded string onto the CLI, there are plenty of people who may want to use Vault + IDP and don't have the level of skill or resources required to do so in a reliable, secure manner.

@jefferai can you provide any additional detail at this time in terms of what level of detail would be abstracted out to plugins? I'm envisioning a plugin would handle two specific calls like:

Is that in line with what you're thinking?

mwitkow commented 7 years ago

@jefferai

If it is the latter, it feels like scope creep entering Generic Auth territory

What is "Generic Auth territory"?

By this I mean: "an auth mechanism that takes a value from a client and returns a token back". I probably made my point badly. What I meant was: What do we define as inputs to this backends? Only Tokens (my preference, but would make the password auth not work)? Or general "credentials input" (passwords, tokens, authcodes)? Because if it is the latter, then it is unclear to how this "pluggable OAuth2 backend" differs from a generic notion of an Auth Backend.

Vault doesn't have a way to display these pages. If you're doing this via textual means I'd be interested in that mechanism. I know the OAuth2 flows well but not the OIDC flows as much.

We were actually doing it for a simple OAuth2 flow, and now are using it for OIDC. You can checkout how Kubernetes does it, very similar to this example code

Basically:

Neither Vault, nor the CLI plugin needs to display any web pages here. This flow has worked incredibly well for our spatial CLI tool, and I believe it is much simpler to implement than server side.

Is this really where you want to go with Vault? I mean, I thought it would be an extensible piece of software with well defined "composable", but "service-oriented" interfaces. Do you think users would trust a dynamically loadable module system?

Doesn't a plugin system add extensibility and composability? Why should users not trust plugins that their administrator has authorized?

Apologies, I should have been clearer here. It's not about the trust in the administrator, it's about whether there is a bondary between a 3rd party plugin and Vault "secrets" or not. In case of "remote API" there is, in case of a dynamic loadable library, there isn't. Even though version 0.1 of a 3rd party plugin may be ok, maybe it's possible that version 0.3 has a vulnerability that compromises Vault as a whole. The administrator wouldn't review the code of each one every time he updates a version.

Since Vault is meant to be the lynchpin of our security system, I would strongly prefer there to not be dynamically loadable 3rd party code in my system. This is an off-topic discussion though, and ultimately it is up to you as maintainers of Vault where you want to take it.

mikeokner commented 7 years ago

at some point the browser redirects back to the "Redirect URL" of the client_id, which brings the browser to the http://localhost:9090/auth/callback which is served by the CLI tool itself.

Right now, Vault's CLI is a very thin client wrapper around Vault's HTTP API. Obviously I'll defer to @jefferai's judgement, but it would seem to be quite a substantial change to the design of the project to start serving HTTP requests from that same CLI. Also, it would then only work when using the bundled CLI and not any other client integrating with Vault's HTTP API unless every client then takes the time to re-implement the same functionality.

No one is suggesting you can't use a client that handles the redirect and then passes the token to the Vault server, just that the server should have the capability to handle the redirect (or other valid oauth2 flows like password) as well.

Since Vault is meant to be the lynchpin of our security system, I would strongly prefer there to not be dynamically loadable 3rd party code in my system.

@mwitkow I suggest taking a look at #2200 and hashicorp/go-plugin as @jefferai suggested. The plugins expose RPC functions running in a completely separate process. There's no dynamic linking or memory sharing. Popular plugins can be shipped built-in with Vault, and external plugins must be whitelisted by checksum by an admin before Vault will load them. I believe your concerns are unfounded.

jefferai commented 7 years ago

Hi @mwitkow ,

Thanks for detailing the flow. I'm kind of torn here, for exactly the reason that @mikeokner specified; putting logic in the CLI means that any other Vault API client can't take advantage of it.

In fact you can imagine that the CLI login helper for this backend could actually take care of the entire flow from the user perspective in just the way you described, except with the actual exchange happening behind the scenes! The CLI helper gets an API client and could call the appropriate Vault endpoint, print out the URL for the user to open (or open it directly if possible), then wait until the flow is complete, finally returning the user's token in the end. But it would still then be a thin wrapper and any other API client or tool could implement the same thin flow.

I do also want to help ease your fears about the plugins. As @mikeokner said, it's a separate process communicating via RPC, but it's actually more complicated than that. We specifically do not want to dlopen, which is one reason we're continuing to use HashiCorp's go-plugin system rather than what's coming in 1.9; we see avoiding cgo as a very good thing (and accommodating database libraries that require cgo is actually a reason we are building the updated database support as plugins in the first place).

Any such plugin that is enabled requires the SHA256 and path to be configured by the administrator; any built-in/included plugin (which is automatically whitelisted) shipped with Vault is actually run by Vault invoking its own binary with the SHA256 and path calculated by Vault when it was originally launched (the path and SHA cannot change during the lifetime of the process without throwing errors).

Once the process is invoked, it is handed a Vault wrapping token; this token contains a unique generated TLS certificate/private key for it to use to talk to the original Vault process. Since it's a wrapping token it's obviously single use. This secures the communication between the plugin and Vault in a method quite similar to how you'd give a wrapping token to e.g. a container to allow it to get its Vault-generated cert.

In the future we'd like to actually create a different mode in go-plugin that maintains the same interfaces but runs the plugin in a goroutine, skipping the need to spawn a process, which would make this flow even better.

So to sum up:

1) Vault remains statically compiled, plugins can be static or dynamic 2) Vault launches a plugin with an RPC interface, no memory is shared outside of IPC 3) Communication between Vault and the plugin is TLS encrypted 4) Plugins require known paths and SHA values 5) Like Vault, plugins support the use of mlock when available.

Hopefully this makes you feel less concerned about the plugin system.

Edit: see 5) above, thanks @briankassouf

jefferai commented 7 years ago

@jefferai can you provide any additional detail at this time in terms of what level of detail would be abstracted out to plugins? I'm envisioning a plugin would handle two specific calls like:

Rather than specific calls, here's an idea of the structs that might be in play:

type OIDCPluginConfig struct {
    // Opaque config values for a specific provider; e.g. for a
    // "generic" OIDC plugin that consumes an already-existing
    // OIDC token being submitted, this may contain items like
    // which claim to use for a unique user ID and which claim
    // to use for group memberships. For a Ping server this may
    // contain items like the API endpoint and the client id/secret.
    Config map[string]interface{}
}

type OIDCPluginLogin struct {
    // Opaque values, e.g. "id_token" or "password"; if a redirect
    // was used, this may contain the returned parameters
    LoginParams map[string]interface{}

    // If a redirect was required, this identifies the state value
    // that can be used to look up original parameters of the
    // request; the presence of this value can be used to
    // trigger appropriate behavior
    RedirectState string
}

// This contains standard values that can be returned from
// the plugin so that all shared backend logic can operate on
// known data members
type OIDCPluginResponse struct {
    // The unique user ID value
    User string

    // The claim that was used for users, for record-keeping/audit
    UserClaim string

    // The groups that the user is a part of
    Groups []string

    // The claim that was used for groups
    GroupsClaim string

    // If a redirect URL is required, the constructed URL
    RedirectURL net.URL
}

In a situation where you already have an OIDC token, you simply pass it in to the plugin, it's validated/parsed, and the user/groups to use are returned.

In a situation where you need to redirect, the login function can instead return a RedirectURL in the reply, which can be returned to the user; when the redirect happens, the state can be sent back to the backend and associated plugin, which can then validate the information (if it's an OIDC token) or fetch UserInfo/an OIDC token (if it's just a bearer token). The client can then retrieve the token. (Ignore the "how does the client do this" part for now as that involves a whole lot of feature enhancements to Vault's core, but they need to be made at some point anyways.) As discussed in the previous post, if the client is a Vault CLI helper, it can do this transparently so that all the user sees is a returned token as normal.

bkrodgers commented 7 years ago

I think the proposal makes sense. In the short term we may need to deploy the code we submitted as a PR, as it sounds like this will be more involved and could be a month or two (or more) away from making it into a release. We're really chomping at the bit to use our oauth2 submission. But my intention would be to go back to the mainline code once you've got something that handles the resource owner flow server side in a similar way to what we submitted. We'd dump the mappings out of our oauth2 backend, remount the new one at the same path, and then load mappings into the new one.

Ideally that transition will be fairly transparent to the users. In fact, we've realized that we don't even need to distribute a custom CLI for this to work. vault auth -method=ldap -path=oauth2 will work just fine. So would vault auth -method=okta -path=oauth2 -username=bkrodgers or vault auth -method=userpass -path=oauth2 -username=bkrodgers, but I like LDAP's implementation better, since it will pull from the USER env var.

I wonder if it'd make sense to have the CLI for auth decouple "method" from the backend in question, and instead have "method" be the mechanism used to gather credentials and pass them to a backend (at any given path, for any given backend that expects those parameters). A lot of the current "methods" don't actually have anything to do with their named backends, and are in fact very similar to other existing ones.

With that in mind, you might only need auth "methods" in the CLI for userpass (Okta, LDAP, userpass, Radius, and OAuth2), token (Github, OIDC, and token), and cert. It looks like the other auth methods don't have CLI helpers at all and instead the docs suggest using vault write to auth, but they could be added if there was a need (I get why there probably isn't for those). Of course you'd leave aliases to the existing ones for backwards compatibility for at least awhile, if not indefinitely, but going forward just maintain "methods" that are truly distinct.

mwitkow commented 7 years ago

@mikeokner @jefferai, thanks for clarifying. It wasn't clear from the code at first glance that there was an RPC boundary. May I suggest starting a /docs/proposals or /docs/internal with a file documenting the plugin system concepts? I think this will massively help in saving time to not have to explain the concepts in details to ignorant users such as myself :)

@jefferai regarding the proposed interfaces, they look ok. Although two things:

From our side, I'll probably take some time later this week to clean up my hacky implementation and we'll try it out in our staging environment as a stand-alone Auth Backend. Similarly to @bkrodgers we need to get the ball rolling and unblock our internal processes. I'd be happy to clean it up (add tests etc.) and submit it as a PR upstream after that. However I don't think I'm well positioned to generalise it to the form of the GenericOAuth2, since I lack the context that you guys have about a) the Vault plugin system b) the other use cases that it should support apart from Tokens (I'm not familiar for example with the requirements of Ping and other proprietary providers).

jefferai commented 7 years ago

@mwitkow You can see info at https://github.com/hashicorp/go-plugin and there's a good talk about it at https://www.youtube.com/watch?v=SRvm3zQQc1Q

@bkrodgers @mwitkow @mikeokner I think there are a couple of moving parts with different schedules here, which is tricky. I think the thing to do in the short term (e.g. ideally this week) is land a design doc that captures the plugin system and describes the interfaces/data structures for it. Revamping the one up above should work fine.

As for the plugins themselves, #2200 hasn't landed yet and I don't want to have a bunch of rewrites if things there need changing; on the other hand I want to make sure anything done for this backend reuses code to the max extent possible, plus has good interfaces, so I will likely be pulling in @briankassouf to help out here in an advisory role. I get the impression though that it should really not be difficult to add the plugins. So that's basically factoring out the Ping-specific and OIDC-specific code into these plugins and leaving most of the rest alone.

The rest of it, for e.g. server-side flows, doesn't have to be done right now, as neither of you need it, although it would be good if we can figure out a reasonable API for the plugins that allows this future capability without much churn in the plugin interface.

So my guess is that since you both already have some implementation work, you'll want to get that going in dev but I don't think it'll be too bad to get it upstreamed over to the united backend and my hope is that it won't be tooooo long before that can be done!

jefferai commented 7 years ago

Hi all,

Traffic died down -- if someone picks this up and updates the doc, do please leave a comment here so that we know it happened and can check!

Josh-West-Bose commented 7 years ago

+1

pidah commented 7 years ago

any movement on this ?

zdoherty commented 7 years ago

+1 also very interested in this

panga commented 7 years ago

+1 (use case: using Keycloak as OpenID provider)

mwitkow commented 7 years ago

@jefferai I finally had the time to clean up and add basic tests to the OIDC provider plugin that I had hanging around on my machine.

The reason for the delay is that we a workaround to not rely on OIDC in Vault just yet, so we managed to push it back by 1-2 months.

Unfortunately, @jefferai, as I said above: I don't think I will have the time (or the knowledge) to work on this to generalize it to the fully pluggable oauth2 provider you envisioned.

I do think, however, that base don feedback from @zdoherty @pidah and @panga a simple (yet still generic) OIDC auth provider will hit most use cases. As such I wouldn't mind following up and making sure that #2796 is a quality PR.