Routing based on HTTP headers, uris and other request properties

[mdemirhan]

Expected Behavior

Today, we support routing only based on the host header. However; we need to support other constructs such as routing based on other http headers, uris, scheme, etc. Istio expose these concepts in VirtualService and Gateway APIs. We need to evaluate the most commonly used ones and expose them in our APIs, rather than asking customer to use a third party solution for that.

Additional Info

Need to do a customer survey before committing to implementing this.

[asciimike]

Copied comments from #67 below...

From a product perspective, I think I'm looking for some higher level constructs that make this easier, maybe like:

domain: mydomain.com
  - path: /login
    service: login  # login.namespace.clusterprefix
  - path: /search
    service: search  # search.namespace.clusterprefix
  - path: /catalog
    service: catalog  # catalog.namespace.clusterprefix

It's possible we might also want to do different subdomains (either for different environments or for regionalization):

domain: eu.mydomain.com
  - path: /login
    service: login-eu  # login-eu.namespace.clusterprefix
  - path: /search
    service: search-eu  # search-eu.namespace.clusterprefix
  - path: /catalog
    service: catalog-eu  # catalog-eu.namespace.clusterprefix

This is somewhat aligned with what Ambassador (API Gateway) does with it's Mapping resource.

Istio and Envoy native options are here:

• Istio Gateway
• Envoy Route

Options as I see them today:

• Require developers use a third party solution (e.g. nginx [low level] or Ambassador [high level])
  • Pro: already built, potentially very easy to understand (e.g. Ambassador Mapping)
  • Con: requires yet another piece of software for developers to understand
• Surface Istio or Envoy configs directly
  • Pro: already built, very powerful
  • Con: potentially confusing for developers looking to get simple things done
• Create our own resource on top of Istio/Envoy configs
  • Pro: likely the fastest solution for developers to get something up and running
  • Con: additional (potentially non-trivial) eng work required to map our format down to Istio/Envoy, likely to diverge from that implementation

Note that I think these solutions exist for a host of other "API gateway" like problems (e.g. data plane auth).

[mdemirhan]

We discussed with @evankanderson and decided to convert this to a documentation effort and not pull this feature in as a first class concept into Elafros. Instead, we will let the customer use any other well known solutions for this. If there is demand in the future to abstract this and make it easier for the customer, we will revisit this.

As part of this work, we will:

• Document setting up a reverse proxy using Istio and GCLB.
• Document using custom header fields or metadata to do routing and traffic splitting.
• Document using and settings up DNS, SSL certs, ...etc.
• Change Elafros controller so that it doesn't override any modifications that the customer makes on Istio configuration (this item should be discussed with @mattmoor for design and implementation)
• (In a separate Github issue): Build an automation tool that can create a YAML file to setup reverse proxy for a given revision/configuration using Istio.

FYI: @vaikas-google, @dewitt, @mattmoor, @mcdonamp

[asciimike]

I don't see how:

(In a separate Github issue): Build an automation tool that can create a YAML file to setup reverse proxy for a given revision/configuration using Istio.

is materially different from:

Create our own resource on top of Istio/Envoy configs

It's just a CRD, right?

[mdemirhan]

The idea behind this tool is to simplify the YAML that needs to be created for Istio. Calling it a tool might be overstating it - it could be as simple as finding the routerule's name for a given revision and creating an istio YAML template where the routerule is pre-populated. This is useful because the users don't necessarily know how a revision related to an Istio RouteRule. Another option would be to expose this information in Revision status.

[asciimike]

This sounds like what the CRD proposed in https://github.com/elafros/elafros/issues/540#issuecomment-382562663 is, right? Do you have an example of what this might look like so we can compare?

[ZhiminXiang]

It should look like the YAML in above comment, but does not need to be a CRD. If we build a CRD for that YAML, then we will always tie the routing with Istio which we may not want. Developers can choose to use the YAML we define and the tool we build if they use Istio. But they do not have to use them. And they do not even have to use Istio since they can choose other proxy they are comfortable with. Building a CRD will give them no choice except Istio.

[mdemirhan]

Yes, it does. Sorry I was confused with something else in the previous comment. This sounds a lot like a Mapping CRD in the comment you have. I don't really know how this should look like. This idea was tossed by @evankanderson to make manually creating Istio rules easier for Elafros users. We haven't thought about it in detail.

[evankanderson]

The difference between a tool for creating an Istio config and supporting a CRD is actually a fairly large:

1. The CRD needs to be able to reconcile updates and report the status of the underlying object; effectively, this is signing up for ongoing updates and "putting things back" when changed.
2. A CRD would limit the known/knowable set of backend targets to a subset of the overall Istio functionality. It would effectively be "an http reverse proxy only for Elafros". By generating an initial Istio configuration, users can then extend that configuration to (for example) add additional pure-kubernetes services, using the Elafros entries as a template. Users can then check in and manage the Istio configuration directly, and they start to learn about the underlying layers of the stack.

I'm concerned that we're trying to build something like Ambassador here without having tried the "buy" approach first. This is clearly a crowded space -- I'd rather provide guidance (with the opportunity to learn where that guidance falls short) about reusing existing solutions which will let us hitch our wagon to the fastest horse.

[mdemirhan]

Closing this as we documented this as part of #924