Basic Envoy Ingress

[andye2004]

In the consul connect docs it says this is on the road map, I know you guys don't generally release details of exactly when something is likely to be released, but I was wondering if someone could comment on if this likely to be near / mid / long term?

I've been using rotor as control plane between consul and envoy in researching the hashicorp stack as part of a mesh POC but this pretty unsatisfactory as it is no longer maintained and is now somewhat behind where we would need it to be as a minimum. I guess I just need to know if as an organisation we would need to invest the time / effort required to build our own control plane.

Thanks, Andy.

[pearkes]

Sorry for the delay here Andy!

Can you describe your use-case a bit more? I think I might know what you mean by edge but want to be sure we understand correctly.

[andye2004]

@pearkes, Thanks for the response. Our intention is to run a multi-node cluster where we will be running a number of services, these services will communicate via envoy sidecars using mTLS in a kubernetes pod like way.

This leaves us requiring an ingress controller so that traffic from outside of the cluster can enter and then be routed on to the services running within the node cluster. So when I say edge what I really mean is an envoyproxy acting as an ingress point to the cluster but which is dynamically configured for L4 and L7 routing by consul connect based on services registered with consul. Ideally, Consul would also be responsible for configuring the certificates on the ingress envoy so that it is capable of using mTLS to route traffic to the envoy sidecars.

Hopefully this fleshes out a bit more on what we are looking for?

[srix55]

Exactly what I am looking for. Looks like Envoy edge doesn't go well with a Consul cluster. As @andye2004 mentioned, rotor seems to be dead. In short, consul can be used with envoy as sidecars for individual services but consul cannot be used with envoy as a front proxy.

[mattmcla]

I'm running into this issue as well. My project would like to accept gRPC traffic from outside the cluster. My first thoughts were to use an envoy front proxy to load balance the requests over a service mesh informed by consul. From what I can tell there doesn't appear to be an easy way to get dynamic configuration from consul to the envoy front proxy. I think envoy should support this via their xDS (etc) discovery api's but there is something missing to link them together.

[ericbrumfield]

I ran into this same issue trying to do an edge front proxy, it seemed like once I configured consul connect with envoy, over on the envoy side and getting xDS setup I saw no way to move forward for a front edge proxy use case and being able to route to the appropriate envoy catalog/service dynamically discovered from consul the way I needed. I definitely felt like I was stuck in a sidecar proxy kind of setup.

I've ended up using traefik for this scenario and overall it has been much quicker and easier to setup than envoy + consul connect. Snapping the discovery from consul catalog was easy and allowed me to direct front edge traffic to the appropriate backend service.

[vasilij-icabbi]

@ericbrumfield did you drop envoy completely or you still used it for sidecar?

[ericbrumfield]

@vasilij-icabbi

For this use case, I totally abandoned envoy and sidecar all together and am currently working on doing it with Traefik, overall in some poc's and for our needs, Traefik is a pretty good fit for a front edge proxy that can pickup services from consul catalog. It also has options for utilizing consul as the control plane in doing a/b or weighted routing via consul catalog tags. I haven't gotten to implementing the a/b or weighted routing to services but based on the documentation it looks like a viable path for us and has been much simpler than envoy for this use case.

[acornies]

Given that Consul 1.6 is out with a bunch of L7 features, how should we proceed replacing something like HAProxy or Traefik as the "edge" router? I watched @nicholasjackson talk at HashiConf '19 which demoed a bunch of new LB-like things, and I'm looking at https://github.com/nicholasjackson/demo-consul-service-mesh but having trouble understanding what portion should replace HAProxy in my case. Is that the right approach?

The "Mesh Gateways" seem like the answer, it's unclear at this point.

[sandangel]

Hi, will consul connect envoy -mesh-gateway solve the problem?

[acornies]

I suppose for now, the simplest explanation is (said in previous comments):

• Use 3rd party load balancer for north-south (client-server) traffic into datacenter, HAProxy, F5, NGINX etc.
• Use Consul Connect envoy integration for east-west (server-server) traffic

The main takeaway for me is that I'm going to leave my HAProxy setup in place, and introduce Connect for my internal traffic only. The new routing features of Consul's central config confused me into thinking it would replace my north-south solution.

Please correct if I'm wrong.

[kgrotel]

Consul has xDS support for Envoy so it should actually be no Problem to setup a dynamic config. I spent the last week trying to make the config work, but the cluster aka Services dont get discovered maybe someone else has got an idea:

node:
  id:                 convoy-front1
  cluster:            convoy-front
admin:
  access_log_path:    /dev/null
  address:
    socket_address:
      address:        0.0.0.0
      port_value:     9901
dynamic_resources:
  lds_config:         { ads: {} }
  cds_config:         { ads: {} }
  ads_config:
    api_type:         GRPC
    grpc_services:
      envoy_grpc:
        cluster_name: infra_consul
static_resources:
  clusters:
  - name:             infra_consul
    connect_timeout:  1s
    load_assignment:
      cluster_name:   infra_consul
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 8502
    http2_protocol_options: {}
  listeners:
  - name:             listen_8080
    address:
      socket_address: 
        address:      0.0.0.0
        port_value:   8080
    filter_chains:
    - filters:
      - name:         envoy.http_connection_manager
        config:
          stat_prefix: ingress_http
          codec_type:  AUTO
          rds: 
            config_source:     { ads: {} }
            route_config_name: n8n
          http_filters:
          - name: envoy.router

[banks]

Hi all, thanks for all of your feedback, just wanted to jump in to say this is certainly on our radar.

There are a few options that exist today and I'll share some rough plans too for the near future.

Our original approach here is that we really don't want to get sucked into building "North-South" edge proxy or API Gateway features into Connect at this point - that's a whole other product line and while it has a lot of overlaps with service mesh (east-west) L7 traffic routing etc, there are also a lot of subtle and not-subtle differences to have a good solution (e.g. public TLS provisioning, User Authentication integrations, different security and traffic logging features etc).

So we aim to provide integrations with existing third-party "API Gateway" products that already have all of those features rather than attempt to compete with them and get pulled into building features that don't benefit the core service-to-service use-case. It's actually pretty easy to integrate just to get a client certificate and get traffic into Connect (L7 stuff is harder) so any proxy developers or people who already use a specific edge proxy and would consider building Connect support as a PR please let us know and we'd be happy to help guide that.

In that vein we've worked closely with the folks at Datawire who have shipped an integrations with Ambassador: https://www.getambassador.io/user-guide/consul/. We know that's Kube-only but we are also working with many more vendors to have close integrations with them.

That said, we've recognized that aligning with eco system partners takes time. In many cases users are needing to bridge the gap a little - not full external "edge" or "north/south" API gateways into Connect, but more like internal gateways for service traffic between newer Connect-enabled in-mesh applications and older legacy systems that are not yet in the mesh for one reason or another.

So to help with both of those cases we are planning two things in the near term (no dates I'm afraid just yet but concrete plans on roadmap):

1. Support for fetching Connect certificates in consul-template. That means it will be possible to easily configure edge proxies like HAProxy and Nginx the same way you always have via consul-template, but have them able to get Connect certs and so send traffic directly to connect-enabled backends without another hop through a side car. This will still leave users free to configure whatever they need for the "frontend" part of the edge proxy of course.
2. A basic Envoy-based ingress proxy that allows you to expose a subset of services in the mesh without requiring mTLS authentication. This is not going to be designed as a full API gateway feature - by default it will just expose those services to anything that can access the proxy with no AuthZ.
   • Basic unathenticated access will work out of the box, but for more advanced control we'll probably allow "escape hatch" style raw Envoy config to be specified that allows injection of customer Filters/TLS config into the "public" listener of the ingress proxy. This means it should be possible to configure custom auth, TLS certs, IP filtering or other rules while benefitting from automatic config for Connect identity and L7 routing to backends. Design TBD - this will be advanced functionality for those familiar with Envoy config but at least provides a way to benefit from the Connect backends without building your own thing from scratch.

Any feedback on this approach is welcome!

[vasilij-icabbi]

Thank you @banks , in the comment here https://discuss.hashicorp.com/t/edge-load-balancing-for-consul-connect/2993/2 person claims that he already does envoy edge proxy using escape hatch, meanwhile you say that you should enable (allow) escape hatch style, could you please explain this piece? Thank you.

[banks]

@vasilij-icabbi great find. The short answer is that there are many levels of "escape hatch" 😄

It's currently possible to replace a pretty large part of the config we send Envoy today and this individual has managed to use that to produce an "edge proxy" setup. The downside is that you have to write a significant amount of Envoy config just to get a basic ingress to work.

In other words, it's possible but hard and not at all a good solution for people right now.

My point 2 above is saying we want to build an easy path to start an Envoy in "ingress" mode which means it will listen without requiring mTLS client auth and will expose a subset of services directly to the outside world. None of that will require any escape hatches at all and should be just as easy as running a sidecar or mesh-gateway today (register the ingress service, one command to start envoy with right config).

The escape hatch part of this new proposal is to allow further customization on top of that basic, unauthenticated ingress. So for people who want to use this as more of an "edge" proxy but need to setup custom certs from a public CA or configure OAuth authentication etc. they will be able to do just those parts with some kind of raw envoy config "escape hatch". But basic unauthenticated ingress will work out of the box.

Does that help?

[vasilij-icabbi]

That would be great to have @banks and Thank you for your explanation, I am just diving into world of service mesh and saw two similar words "escape hatch" in different context :)

Personally I think this part of functionality described by you above and in point 2 above of the above is very much missed and forces people to switch to different solutions, or add more tools to their toolbox. I am for point 2, I would create feature request but doubt have enough knowledge to describe properly.

[banks]

No need for a feature request thanks - this issue is one and we've already planned to work on it!

On Tue, Oct 8, 2019 at 2:24 PM vasilij-icabbi notifications@github.com wrote:

That would be great to have @banks https://github.com/banks and Thank you for your explanation, I am just diving into world of service mesh and saw two similar words "escape catch" in different context :)

Personally I think this part of functionality described by you above and in point 2 above of the above is very much missed and forces people to switch to different solutions, or add more tools to their toolbox. I am for point 2, I would create feature request but doubt have enough knowledge to describe properly.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hashicorp/consul/issues/5695?email_source=notifications&email_token=AAA5QU4UMDYVN2XNUMIFLUDQNSCX7A5CNFSM4HHIXPQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAUEVLQ#issuecomment-539511470, or mute the thread https://github.com/notifications/unsubscribe-auth/AAA5QUZS6RSPCDM2QBVVHNDQNSCX7ANCNFSM4HHIXPQQ .

[banks]

Hi @andye2004 I hope you don't mind but I've renamed the issue slight to match the proposal above so we can use it to track progress on this feature going forward instead of creating another issue without the context.

[andye2004]

No issues here @banks

[eikenb]

For anyone here interested, with release 0.23.0, consul-template now supports fetching the connect root and leaf certificates as well as the connect enable services. There are simple introductory documents describing how to create ingress gateways with NGINX and HAProxy as well.

[Gufran]

For anyone who comes looking for this, I've created an xDS server that lets you deploy Envoy as an edge proxy for Connect enabled services. The mechanism is simple, the server registers itself in consul catalog as connect native service and shares its leaf certificate with Envoy. Envoy does the rest.

You can read more about it at https://docs.flightpath.xyz/ and provide feedback if you end up using it.

[blake]

Hi folks,

We just announced the beta release of Consul 1.8 which now includes this basic Envoy ingress. See the announcement blog for more details (https://www.hashicorp.com/blog/announcing-hashicorp-consul-1-8/), or head over to https://www.consul.io/docs/connect/ingress_gateway to see the docs for configuring a gateway.

We look forward to hearing your feedback on this new feature.

[blake]

As of yesterday Consul 1.8 is now GA. We've published a couple of blogs and guides on configuring Ingress gateways.

• https://www.hashicorp.com/blog/ingress-gateways-in-hashicorp-consul-1-8
• https://learn.hashicorp.com/consul/developer-mesh/ingress-gateways

I am closing this issue since this feature has now been implemented. Please feel free to reach out on discuss.hashicorp.com if you have further questions about configuring and using ingress gateways.

[savaskalyon]

Does this ingress proxy support grpc services as backend like other ingress proxies, e.g. traefik, envoy edge proxy etc?

[blake]

Does this ingress proxy support grpc services as backend like other ingress proxies, e.g. traefik, envoy edge proxy etc?

Hi @savaskalyon, support for gRPC just added in https://github.com/hashicorp/consul/pull/8458 and will be available in the next Consul 1.8.x release.