Expose administrative APIs at a different port (e.g. 4445)

[aeneasr]

You probably all know that the internal access control engine for ORY Hydra has moved to other projects. The reasoning was, that getting started with the technology was really difficult, you had to deal with root clients, administrative policies, OAuth2 scope, ...

So far, the feedback has been stellar. Everyone enjoys less friction and getting started quickly. There's however a catch to this as well. If you run ORY Hydra in an environment (e.g. Heroku, CloudFoundry, ...) where you don't have access to an API Gateway and where you don't want to customize the Docker Image in order to add an API Gateway, then you will have a hard time protecting the administrative endpoints.

One idea that is common with other services is to expose these APIs at a different port. For example,

• /clients would run on port 4445
• /oauth2/auth would run on port 4444

That way it (hopefully) becomes easier to perform access control in these environments. An obstacle however is, how do we define administrative APIs? The /oauth2/introspect endpoint does have access control (you need a valid access token), but it's also a priviledged endpoint (typically for resource servers). You might want to expose this endpoint however to the open internet. So, does this run on 4444 or 4445?

It could also lead to serious confusion when trying to perform requests and it could lead to complicated CLI set ups like

# The introspection needs to first request an access token (port 4444), then perform introspection (port 4445)
hydra introspect
  --frontend-url http://localhost:4444/
  --admin-url http://localhost:4445/

which is just terrible.

Another option is to extend the ORY Oathkeeper functionality and have (prebuilt) docker images that make this easy to set up and deploy to these platforms.

I'd really like to hear what you think about this! This issue is blocking for 1.0.0.

[aeneasr]

There was a discussion in the chat, summary is:

1. It's probably a good idea to separate those endpoints
2. There could be a distinction between "user-facing" and "admin-facing"
   1. "user-facing": Endpoints that perform the oauth2 flow, such as /oauth2/auth, /oauth2/token, /oauth2/revoke, /userinfo and the ./well-known ones.

The /oauth2/introspection endpoint is defined for internal use (resource server) and requires authorization. How that authorization looks like is out of scope:

To prevent token scanning attacks, the endpoint MUST also require some form of authorization to access this endpoint, such as client authentication as described in OAuth 2.0 [RFC6749] or a separate OAuth 2.0 access token such as the bearer token described in OAuth 2.0 Bearer Token Usage [RFC6750]. The methods of managing and validating these authentication credentials are out of scope of this specification.

So one point raised is that exposing this endpoint at a clearly defined "admin" port would already imply some type of authorization ("priviledged port").

In general, the proposed change was viewed positively.

[MOZGIII]

I personally like this idea, however I'd propose something a bit different. In my opinion, if it's possible, we should separate OAuth and Hydra-specific API, not the user-facing or admin-facing ones. Rationale is the following: hydra should probably only expose OAuth API to non-admin consumers, and all hydra-related (custom) APIs can just be considered privileged. In our case, that access to hydra-specific endpoints will be organized via backend calls.

Also, note that the spec does not explicitly state that introspection is intended for internal use. All it says is there should be some kind of authorization. I would propose implementing two authorization options there (for user to choose): either a secret bearer token, or an plugable authorization for incoming request via webhook sent to something else, containing request details and requiring whether to allow access or to deny it (with some kind of message) in response.

For admin-level access, I think there should be a separate endpoint (not even under /oauth resource), that doesn't necessarily follow the OAuth spec, but provides the introspection information in the way it makes the most sense for hydra. What I mean by that is hydra most probably is capable of providing more data on it's internal state to an internal consumer than it would want to provide via public API. Format for that data might be different from what the OAuth spec dictates if makes more sense for hydra to deviate there, simply because this API should be a custom hydra endpoint (since it lives on a "privileged control listener").

[MOZGIII]

Btw, take a look at what Kubernetes does to authorize access. It has a lot of nice patterns. Hydra could make use of few simple ones to authorize access to /oauth2/introspection.

[androbi-com]

My guess is that a protection based on endpoint url paths (done by the user) would be more flexible. How about an example setup of an nginx reverse proxy to go with hydra? I'd personally prefer this. Would it be done in a way that user and admin endpoints can be mapped to the same port?

[aeneasr]

@clausdenk yes that would be the current approach. We haven't added an ngnix example just yet but planned to do so. Depending on the consensus in this issue we'll move ahead (or not) with this.

@MOZGIII thank you for your feedback. I think the OAuth 2.0 Token Introspection endpoint is actually the only controversial one. All other endpoints (/oauth2/flush, consent, client, jwk) are no-brainers when it comes to separating between consumer/user-facing ("unpriviledged") and internal-facing ("priviledged"). Currently, that endpoint is protected by requiring an OAuth 2.0 Client (id+password) or a valid access token. Obviously, that's not very secure because I, as a user, can issue myself an access token and query that endpoint.

I also agree with the differentiation between internal session state and public session state. Currently, the introspection endpoint is seen as an elevated endpoint that exposes (potentially) internal session state. It's really up to the developer what data get's stored there - but the possibility of accidentally exposing internal session state is there. The downside of adding another endpoint is one of complexity. I think we should reduce complexity (in terms of learning curve) as much as possible to make adoption as easy (and hopefully secure) as possible. By - for example - exposing the introspection endpoint on the secondary (admin) port, you - the developer - would have to make the concious choice of exposing that particular port+path to the public net (and hopefully not store sensitive data there).

I think re-adding authorization is counterproductive. We got rid of it to reduce complexity when developing and authorization can easily be added with ORY Oathkeeper (as shown in ory/examples). Having said that, I have not yet looked at the kubernetes docs but will definitely do that in the coming days!

So far, thank you both for participating and your input. It is very valuable. I hope we can find the best path forward!

[MOZGIII]

As a service, Hydra must provide it's own authorization for that endpoint (endpoint MUST also require some form of authorization to access this endpoint). What that means in the spec is Hydra can't rely on being deployed behind a reverse proxy that would manage the access control. If the authorization via a reverse proxy makes sense, hydra should probably provide the ability to disable the internal authorization.

[MOZGIII]

I think re-adding authorization is counterproductive. We got rid of it to reduce complexity when developing and authorization can easily be added with ORY Oathkeeper (as shown in ory/examples). Having said that, I have not yet looked at the kubernetes docs but will definitely do that in the coming days!

True, but just partially. In my opinion, what hydra needs is a very simple but pluggable way to authenticate requests, and afaik it's only required for introspection requests. So, definitely re-adding the complexity of built-in client-based authorization doesn't make sense. That effectively means that Hydra, being an OAuth server, also becomes it's own OAuth client, which doesn't look good to have built in - I'd rather implement that as a introspection auth webhook handler.

[aeneasr]

It depends, the wording is intentionally vague - authorization can be obtained via different measures - be it access to the internal network, being deployed behind a reverse proxy, requiring valid access credentials. We're currently implementing the examples raised in the spec (securing the endpoint via access token/client credentials) but this doesn't - imo - provide any typo of security other than you - the attacker - requiring an access token to perform token scanning attacks.

That being said, the use case of To prevent token scanning attacks is a very narrow one. IMO it's impossible to guess, in the lifetime of planet earth, an access token issued by ORY Hydra. So there is really no need to defend against scanning attacks. Another (imo) inconsistency is that you can perform token scanning attacks by just making requests to your API with random access tokens. Like - what's the difference?

Further down in the spec, there's a bit more context on this:

If left unprotected and un-throttled, the introspection endpoint could present a means for an attacker to poll a series of possible token values, fishing for a valid token. To prevent this, the authorization server MUST require authentication of protected resources that need to access the introspection endpoint and SHOULD require protected resources to be specifically authorized to call the introspection endpoint. The specifics of such authentication credentials are out of scope of this specification, but commonly these credentials could take the form of any valid client authentication mechanism used with the token endpoint, an OAuth 2.0 access token, or other HTTP authorization or authentication mechanism. A single piece of software acting as both a client and a

protected resource MAY reuse the same credentials between the token endpoint and the introspection endpoint, though doing so potentially conflates the activities of the client and protected resource portions of the software and the authorization server MAY require separate credentials for each mode.

[aeneasr]

So according to that, the client making the request should be authorized to do so - this was previously handled with our access control policies. In turn, this implies that the current measures deployed (active token) are insufficient.

On possibility to solve this easily would be to require a specific scope (such as introspection) to be granted when making the request..but man..this will be hard to teach people.

[MOZGIII]

@arekkas good point about the spec scanning attacks, I don't think we understand the attack vector here. We should ask the guys who wrote the spec for advice, can you contact them?

[MOZGIII]

On possibility to solve this easily would be to require a specific scope (such as introspection) to be granted when making the request..but man..this will be hard to teach people.

I don't see any how it's hard to grasp :)

[MOZGIII]

For my use case, I'd only use the privileged (admin) variant of the introspection endpoint, so just the ability to disable the publicly accessible OAuth variant of token introspection would be required. I don't want to set up a reverse proxy just for that.

[aeneasr]

@arekkas good point about the spec scanning attacks, I don't think we understand the attack vector here. We should ask the guys who wrote the spec for advice, can you contact them?

I think it's clear. They require authorization in order to prevent token scanning. I think the example of using access tokens is sorry trash, because - well - you have to have a valid access token to perform the request. And that way you can actually scan for tokens. So really no idea what type of safeguard this is.

The error message is 401 or 403 if the token from the authorization header is invalid and 200 if it's valid. The response payload allowed: ... depends on the token from the body.

Maybe avoiding this BS and going with a more sane approach (such as declaring this an administrative API) will save us from going insane here :D

[aeneasr]

By the way, I couldn't find contact information on the authors, but I'll try to talk to them. Usually they hide this info so they don't get constantly spammed by consultants and head hunters, so it might be difficult to come around.

[MOZGIII]

@arekkas I can think of ways the security might be compromised in our system through token scanning if some other security layers are breached. Though, not brute-force scanning.

[MOZGIII]

Maybe avoiding this BS and going with a more sane approach (such as declaring this an administrative API) will save us from going insane here :D

This works for my use case, not sure about others. I'd say if it passes the OAuth compliance we're good.

[aeneasr]

Ok, I still think this is a good idea. The design would look as follows:

• hydra serve admin: Exposes all administrative endpoints. This includes:
  • All /clients endpoints. I am aware that /clients implements OpenID Connect Dynamic Client Regsitry which (sort-of) allows public access but since we do not implement any type of access control, anyone could modify existing clients which is why this should not be a public endpoint.
  • All /jwks endpoints
  • All /health, /metrics, /version endpoints
  • All /oauth2/auth/requests endpoints
  • /oauth2/introspect as OAuth 2.0 Token Introspection is reserved for resource servers - exposing this endpoint might compromise internal session state which is why we play it safe and have it at the admin port.
  • /oauth2/flush
• hydra serve public exposes:
  • /oauth2/fallbacks/consent
  • /oauth2/fallbacks/error
  • /oauth2/auth
  • /oauth2/token
  • All ./well-known endpoints
  • /userinfo
  • /oauth2/revoke

All environment variables are shared across the two commands (serve admin/serve proxy) and you can choose the port with PORT. This will obviously also change how we use --endpoint-url and HYDRA_URL in the CLI, most likely by keeping these flags and, where neccessary, replace them with --admin-url and --public-url (although I don't think that that's actually neccessary).

[aeneasr]

Note to self: Need to make sure that there are no race conditions when starting a new hydra installation on an empty database regarding JWK creation

[MOZGIII]

This looks good, when will it be available?

[aeneasr]

It's a lot of work, can't say honestly.

[davidf-uc3m]

I prefer require an "introspect" scope and keep the endpoint open. Brute force attack could be done only using valid credentials. Brute force attacks can be detected by other means that could lead to invalidate those client credentials. Keep in mind that some Resource Servers could be a third party api developed outside validateing scopes from tkoens issued by our Hydra. Those 3rd party apis should be able to call introspect endpoint.

[aeneasr]

An OAuth2 scope is not a protective mechanism. OAuth2 Clients can (if OpenID Connect Dynamic Client Registration is open) define their own scopes.

The main reason for putting OAuth2 Introspection at another endpoint is:

This specification defines a method for a protected resource to query an OAuth 2.0 authorization server to determine the active state of an OAuth 2.0 token and to determine meta-information about this token. OAuth 2.0 deployments can use this method to convey information about the authorization context of the token from the authorization server to the protected resource.
- Source

When we are talking about protected resources, it usually implies "resource servers" which are first-party systems running within the infrastructure in which the Authorization Server is available as well.

While it could be possible to expose this functionality to other parties over the open internet, it's not the primary use case, which is why - imo - this makes sense to keep at the "internal" or "admin" service.

What is your use case for exposing this functionality?

[aeneasr]

Here's also some more information on that and I would also like to note that - to my knowledge - no popular API system that uses OAuth2 (Facebook, Google, Dropbox, GitHub) exposes the OAuth2 Introspection endpoint.

[davidf-uc3m]

Thank you for the link. I'll review it more concisely.

[aeneasr]

Also keep in mind that exposing this functionality on the "internal" port will not prevent you from actually exposing this endpoint to the public internet. You, as the operator, however have to make the conscious decision to do so, which I think is the right way to approach it!

[davidf-uc3m]

Ok. In case the introspect endpoint were deployed as administrative endpoint.....no extra security would be checked by Hydra (neither basic auth nor bearer)....Am I right?

[aeneasr]

That's a good question which I have thought about for a while. I believe we would keep the current authorization system in place as it's (sort of) what the spec suggests (note - doesn't require that) and it would make exposing that endpoint easier.

[davidf-uc3m]

And requiring "hydra.introspect" as scope? I think it's a good approach, this way it forces (as now does) to register the resource server as hydra client (in order to keep all resource servers tracked)

[aeneasr]

See my previous comment:

An OAuth2 scope is not a protective mechanism. OAuth2 Clients can (if OpenID Connect Dynamic Client Registration is open) define their own scopes.

The OAuth 2.0 Scope only "scopes" what a someone allowed a token to do, not what a client is allowed to do. That's a huge difference. If you use basic authorization at that endpoint, there is no OAuth2 scope. OAuth2 Scope is not an access control mechanism like RBAC, ACL, and so on.

[davidf-uc3m]

Ok, it's true.

[decoomanj]

Why don't you expose the path on the public AND the admin part? On the public one, the client needs clientID/secret (as the specs suggest), and on the admin part there is no security necessary (since this would be a Hydra thing).

[aeneasr]

That's a fresh idea! I still have some reservations:

By exposing introspection on the admin port, you, the developer/operator makes the conscious decision to expose that endpoint to the public internet by setting up some routing. I think this is important because token introspection exposes the internal state/data of an access token. Assuming you, the developer, chooses to include personal or sensitive data personal or business in the access token (email, profile picture url, system privileges, ...), the payload would be accessible to the public internet by simply having an OAuth2 Client (which is registered at e.g. my-company.com/developers) capable of introspecting the token. While this might be obvious to some, it might also be an "oh shit" moment for others.

While I really like the thinking here, I believe that we should limit introspection to the admin/privileged port and remove the HTTP Authorization requirement. Exposing that functionality is then a decision with trade-offs (no sensitive session data) that must be made. This also allows you to set up access control the way you like it (e.g. access tokens with Oathkeeper or whatever).

What do you think?

[decoomanj]

Normally seen, nobody will expose the admin-interface over the API-gateway unless they really know what they are doing. In our case, only other internal services will communicate with Hydra (just as they do with Cognito). I don't see a reason why an admin-endpoint would ever be exposed without an additional layer protecting it.

I think it would be ok when the public introspection endpoint would be exposed. It is protected by default and adheres to the spec. I understand that there is a risk involved here too, but if you strive for your own OIDC certification, you may probably need this?? Anyway, the admin introspection endpoint is protected in that sense that it is only accessible from within your own network.

The fact that you expose sensitive data is in the nature of OIDC. You documented it already well in the code.

For my own use-case, I would use the (unprotected) admin-introspection first, and then see if my clients are in need of the public endpoint. When yes, I will expose it on the gateway.

But maybe there is also another difference between public and admin. When my gateway wants to validate a token, it will send it to the introspection endpoint. There will be no clientID or whatever there. The gateway just needs to know if the token is valid.

On the public endpoint, the client can probably only check the validity of tokens issued to him, right? This would make a subtle difference IMHO.

[MOZGIII]

I think, intuitively it makes sense to only expose introspection endpoint on the admin port. However, OIDC certification requires this endpoint, and it requires it to be protected. And not exposing it on the public port may be considered an the "lack" of endpoint, despite it being exposed on the admin port (meaning that move is not considered "protected"). So, now, there should be a certification committee to judge what's the case. However, if we expose the endpoint on both endpoints - with protected version on one of them - it will completely eliminate the uncertainty there. Due to that, I'd just expose on both.

By the way, if we have two endpoints, we can actually mark some scopes to be hidden on the public introspection endpoint - and this may be an additional feature.

[davidf-uc3m]

I agree: It should be exposed in both ways.

[aeneasr]

I think intuitively it makes sense to only expose introspection endpoint in the admin port. However, OIDC certification requires this endpoint, and it requires it to be protected.

That is not true, OpenID Connect spec does not care at all about this endpoint nor is it required for certification. The endpoint isn't even a valid option in OpenID Discovery. Please clarify your source.

[MOZGIII]

@arekkas I'm not sure about that, this is my understanding from the discussion here. I'll try to look it up.

[aeneasr]

@arekkas I'm not sure about that, this is my understanding from the discussion here. I'll try to look it up.

If there is no source, then this is false. I was personally involved in the certification process, signed the documents, reviewed the test samples, and even wrote most of the stuff required for it. I can tell you with 100% confidence that introspection is in no way affiliated with the certification process. This should already be clear as introspection is an IETF specification and not one from the OpenID Foundation. They lack the authority to certify that.

[decoomanj]

Ok, this was a false assumption from my side then (probably mixed it up with userinfo which is in well-known). Nevertheless I think exposing it on public and admin makes sense. Then it is easy to expose it to clients when needed, and you can use the admin safely in your own environment.

[MOZGIII]

@arekkas sorry for confusion, you can ignore that part of my message :)

[aeneasr]

tl;dr OAuth2 Introspection will be stripped of access control and moved to privileged port. It might be (re-)added to the public port with access control at a later stage.

Long version:

We talked this through internally and came to the conclusion that the endpoint will be exposed at the privileged port without access control for beta.8 and probably also rc.1. There is still the possibility to add this endpoint with access control to the public port, but we need more time to make a good decision (and design) here and we also want to see if it is actually necessary for us to do so (maybe developers don't expose the endpoint to the public at all and the feature would be unused).

Here's the current draft for the changelog:

### Split of Public and Private Ports

Previously, all endpoints were exposed at one port. Since access control was removed with version 1.0.0, privileged
endpoints (JWKs management, OAuth 2.0 Client Management, Login & Consent Management) were exposed and had to be secured
with sophisticated set ups using, for example, an API gateway to control which endpoints can be accessed by whom.

This version introduces a new port (default `:5555`, configurable using environment variables `PRIVILEGED_PORT` and
`PRIVILEGED_HOST`) which is serves all privileged APIs:

* All `/clients` endpoints.
* All `/jwks` endpoints.
* All `/health`, `/metrics`, `/version` endpoints.
* All `/oauth2/auth/requests` endpoints.
* Endpoint `/oauth2/introspect`.
* Endpoint `/oauth2/flush`.

The second port exposes API endpoints generally available to the public (default `:4444`, configurable using environment
variables `PUBLIC_PORT` and `PUBLIC_HOST`):

* `./well-known/jwks.json`
* `./well-known/openid-configuration`
* `/oauth2/auth`
* `/oauth2/token`
* `/oauth2/revoke`
* `/oauth2/fallbacks/consent`
* `/oauth2/fallbacks/error`
* `/userinfo`

The simplest way to starting both ports is to run `hydra serve`. This will start a process which listens on both ports
and exposes their respective features. All settings (cors, database, tls, ...) will be shared by both listeners.

To configure each listener differently - for example setting CORS for public but not privileged APIs - you can run
`hydra serve public` and `hydra serve privileged` with different settings. Be aware that this will not work with `DATABASE=memory`
and that both services must use the same secrets.

### OAuth 2.0 Token Introspection

Previously, OAuth 2.0 Token Introspection was protected with HTTP Basic Authorization (a valid OAuth 2.0 Client with
Client ID and Client Secret was needed) or HTTP Bearer Authorization (a valid OAuth 2.0 Access Token was needed).

As OAuth 2.0 Token Introspection is generally an internal-facing endpoint used by resource servers to validate
OAuth 2.0 Access Tokens, this endpoint has moved to the privileged port. The specification does not implore which
authorization scheme must be used - it only shows that HTTP Basic/Bearer Authorization may be used. By exposing this
endpoint to the privileged port a strong authorization scheme is implemented and no further authorization is needed.
Thus, access control was stripped from this endpoint, making integration with other API gateways easier.

You may still choose to export this endpoint to the public internet and implement any access control mechanism you find
appropriate.

If there are any doubts or concerns left, please raise them now!

[aeneasr]

Oh, and please let me know if you think that it should be called "privileged port", "internal port", "admin port", or any other name suggestions.

[MOZGIII]

I'd suggest using server instead of port. In addition to that: public server and control server. What you think?

[aeneasr]

I really like control, and yes definitely server - I was referring to the environment variable PUBLIC_PORT but that was obviously out of context :D

So we would have environment variables PUBLIC_PORT, PUBLIC_HOST, CONTROL_HOST, CONTROL_PORT and commands hydra serve (runs both servers), hydra serve public (runs only public server), hydra serve control (runs only control server).

Opinions?

[MOZGIII]

That's great, flexible enough for me!

[MOZGIII]

Can we also expose /health when using hydra serve public?

[androbi-com]

I guess you want hydra serve to serve both for backward compatibility? With this in mind it would seem more natural to use hydra serve -only-public or whatever for reduced setups.

[aeneasr]

Yeah I think it makes sense to expose /health/* (but not /version, /metrics/*) at the public endpoint too. Load Balancers will probably want to check the health of each service individually.

[aeneasr]

Hm, what is the difference between hydra serve public and hydra serve --only-public? Aren't both conceptually the same (you reduce the scope of the command) - whilst one is using a flag and the other a subcommand?

[androbi-com]

With hydra serve public and hydra serve control I would probably expect hydra serve public control to give the full server and hydra serve to throw an error (missing arg). With "only" you express that the additional arg is a reduction of the full command.