Expose administrative APIs at a different port (e.g. 4445)

[aeneasr]

You probably all know that the internal access control engine for ORY Hydra has moved to other projects. The reasoning was, that getting started with the technology was really difficult, you had to deal with root clients, administrative policies, OAuth2 scope, ...

So far, the feedback has been stellar. Everyone enjoys less friction and getting started quickly. There's however a catch to this as well. If you run ORY Hydra in an environment (e.g. Heroku, CloudFoundry, ...) where you don't have access to an API Gateway and where you don't want to customize the Docker Image in order to add an API Gateway, then you will have a hard time protecting the administrative endpoints.

One idea that is common with other services is to expose these APIs at a different port. For example,

• /clients would run on port 4445
• /oauth2/auth would run on port 4444

That way it (hopefully) becomes easier to perform access control in these environments. An obstacle however is, how do we define administrative APIs? The /oauth2/introspect endpoint does have access control (you need a valid access token), but it's also a priviledged endpoint (typically for resource servers). You might want to expose this endpoint however to the open internet. So, does this run on 4444 or 4445?

It could also lead to serious confusion when trying to perform requests and it could lead to complicated CLI set ups like

# The introspection needs to first request an access token (port 4444), then perform introspection (port 4445)
hydra introspect
  --frontend-url http://localhost:4444/
  --admin-url http://localhost:4445/

which is just terrible.

Another option is to extend the ORY Oathkeeper functionality and have (prebuilt) docker images that make this easy to set up and deploy to these platforms.

I'd really like to hear what you think about this! This issue is blocking for 1.0.0.

[aeneasr]

Ah, that makes total sense, good point. Maybe hydra serve public control isn't such a bad idea? :D

[MOZGIII]

I like two separate commands more. Either that, or flags, i.e. with --.... Having subcommands used in that manner is confusing.

[androbi-com]

@arekkas But in this case older Dockerfiles (that use hydra serve) will stop to work.

[aeneasr]

@arekkas But in this case older Dockerfiles (that use hydra serve) will stop to work.

That's fine for two reasons:

• Our old Dockerfiles available on Docker Hub will still work as the software did not change, and the new Docker Images will work as well as we'll update the default entrypoint.
• We're still in beta which has been communicated as "good enough to work with and potentially expose to the public" but also with the clear warning of "we might break bc from version to version until we're stable".

A breaking change from versions <= 1.0.0 happened anyways as the server command was hydra host back then.

In general I'd like to make this clean and understandable with the stable release than keep BC between betas :) I hope you agree!

[MOZGIII]

@arekkas there still will be just a single binary with everything, right? I make use of both bare variant of the docker image and alpine variant btw, kudos for having two images.

[aeneasr]

Yes, one binary

[MOZGIII]

@clausdenk hydra server will still work, it will just listen on two ports instead of one. Two new subcommands will allow starting either of the servers separately.

[androbi-com]

I find hydra serve public control a bit long for what is probably the default action, but it is expressive. I can imagine two possibilities, one reductive, one additive:

hydra serve + hydra serve --only-public + hydra serve --only-control

hydra serve all (or hydra serve public control) + hydra serve public + hydra serve control

But the syntax should express if we reduce or add. Agree 100% with breaking and getting a good expressive syntax.

[aeneasr]

The all option is what we have in place for oathkeeper at the moment (oathkeeper serve proxy, oathkeeper serve api, oathkeeper serve all) so it wouldn't be a completely new concept to the ecosystem. I agree that chaining the serve processes is a bit lengthy. My personal opinion is towards serve all|public|control where serve all could be an alias for serve public control.

[androbi-com]

Sounds good then. It also is more flexible in case you split another port :)

[MOZGIII]

I'd suggest using subcommand flags instead of subcommand arguments, because arguments are usually positional, while flags are not. In practice it only matters a little, and I'm ok with whatever we'll settle on. Examples: hydra serve --all, hydra serve --public, hydra serve --control Maybe even hydra serve -cp instead of --all.

[aeneasr]

Arguments are easier to handle when it comes to help messages - hydra help serve public can (can, not must) have a different message than hydra help serve control while hydra serve --all -h will have the same message as hydra serve --control -h. I think that allows for a bit more flexibility here :)

[aeneasr]

Oh, and it's easier to spot in the overview, see hydra help clients for example:

hydra help clients
Use this command to create, modify or delete OAuth2 clients.

Usage:
  hydra clients [command]

Available Commands:
  create      Create a new OAuth2 client
  delete      Delete an OAuth2 client
  get         Get a client by id
  import      Import clients from JSON files

[MOZGIII]

That's ok, I just don't like public control thing. But whatever, I've seen worse and I'm still alive :)

[boydgreenfield]

@arekkas I like cleanly separating privileged vs. public endpoints. One feature that would make hydra much more usable out-of-the-box would also be supporting basic auth protection for the privileged endpoints, e.g., via a PRIVILEGED_BASIC_AUTH_USER and PRIVILEGED_BASIC_AUTH_PASSWORD environment variable or flag.[1]

For context, we've deployed Hydra to a PaaS (Aptible, but this would apply to Heroku or most others) and unfortunately couldn't use the direct Hydra Docker images because these endpoints are unprotected and we're not using an API gateway (nor is that easy to do/common with those setups).[2]

In many PaaS environments, this addition would let users get set up much more quickly in a secure way.[3] Just my two cents. Thanks for all the great work on this project! 👏

---

[1] I realize this is a slippery slope towards requiring a multitude of auth schemes... But the alternative is requiring users without API gateways to roll their own in order to deploy Hydra, so built-in basic auth seems like a pretty good fallback option.

[2] Our solution was to set up an NGINX proxy with basic auth for all of the privileged endpoints. Separating the endpoints is a big win as we don't have to keep the list of routes in sync (just protect one port and not the other) but it'd still be very nice to be able to use the ory/hydra Docker images directly vs. re-packaging and increasing the friction around updates.

[3] Ps. One other note is that many of these environments also handle TLS termination. May be worth noting in the docs that this requires the --dangerous-force-http flag and that's it OK in that context (it's a reasonably, but very alarming flag name :smile:).

[MOZGIII]

Side note: I'm not sure, but I think dangerous-force-http is not supposed to be used in the scenario when TLS termination is done externally.

[boydgreenfield]

@MOZGIII Well, glad to hear that now as we're only running it in a staging environment (and it's on a separate network)! Do you know what one is supposed to do? Without that flag, the server was just refusing all of our connections (but maybe I didn't set the HYDRA_URL correctly?)...

[aeneasr]

@boydgreenfield thank you for the detailed feedback. Protecting the control/privileged/admin service is definitely something that should be easy to set up. Heroku is a bit of an outlier as they do not expose any functionality of the load balancer whatsoever. Other platforms (GCP, AWS) support - for example - TLS Client Certificates in their LBs or API Gateways. Nevertheless, this should be more easy to set up. Maybe we could provide a docker image that runs Oathkeeper alongside Hydra privileged and that requires - per default - some type of basic authorization. This is what we're trying to get working with the examples repository, but we're not there yet.

I think adding it to the core would decrease flexibility and also raise questions of permission systems (RBAC, ACL) which we want to avoid.

Regarding -dangerous-force-http, the best was to approach it is to remove the flag and set environment variable HTTPS_ALLOW_TERMINATION_FROM. This allows you to set an IP range (CIDR notation) which allows requests coming from that range to be HTTP instead of HTTPS. Please note that hydra expects HTTP Header X-Forwarded-Proto to be set to https for all incoming requestst. Most routers/gateways support that, some don't. The header transports (similar to X-Forwarded-For) the original scheme that was used when calling the gateway/edge.

[boydgreenfield]

@arekkas OK thanks. I'm not 100% certain if we are guaranteed to know the IP range for HTTPS_ALLOW_TERMINATION_FROM to work properly but will investigate.

Regarding basic auth, that seems reasonable – I understand not wanting to clutter the core, especially since you have a separate service for this. The only downsides are, of course, that you end up then needing to run Oathkeeper for a very limited subset of its functionality and you end up with a 2 process Docker container (begging some questions about health checks, etc. on a PaaS). It'd certainly be nice to just be able to use a TLS Client Cert! 😄

Edit after seeing @MOZGIII's comment: Regarding TLS, I just meant it'd be nice if our load balancers supported them; this is not a request to build that into Hydra!

[MOZGIII]

I like TLS client cert idea too. However it's unlikely I'll need this feature - when we deploy local PKI it's almost always handled with local loadbalancers. For that case, btw, unix socket support would be useful.