Guidance on Ingress?

[krancour]

Invariably, some services need to be exposed to the outside world. In k8s, without Consul Connect in the picture, I'd obviously tackle this using Ingress resources and an Ingress controller, but it's not at all clear to me how I can make those work nicely with Consul Connect in the picture.

I've just made a succession of attempts to make Consul Connect and Ingress work together. They've looked something like this:

1. Create k8s Service resource for application and an Ingress resource that references it. This doesn't work because the application is listening on 127.0.0.1, so only traffic that arrives via proxy gets in.

2. Make consul sync its services to k8s. This results in a service for my app in Consul's namespace, which wasn't what I'd have hoped for, but ok. The service is of type ExternalName. So I made an Ingress in that same namespace that references that service... no dice. My ingress controller doesn't resolve the backend endpoints.

3. Figuring the previous failure is because my ingress controller wasn't, itself, Consul-aware, I annotated it to enable proxy sidecar injection. Now I'm imagining that the service should resolve, btu it still doesn't. Maybe it's a DNS issue?

I'm left scratching my head at this point, wondering how an Ingress controller and Consul Connect are supposed to work together. Or aren't they?

fwiw, I notice that in the da-connect-demo repository "simple_kube" example, the "ingress" isn't an Ingress controller at all. It's a one-off instance of Nginx with customer configuration (i.e. nginx.conf). That seems hacky and doesn't seem like it would scale well or be very convenient to have to repeat for every service I want to expose to the outside world.

So... is there any guidance on the right way to do this?

[patoarvizu]

Here's what I did to make this work (hopefully I didn't miss any steps but feel free to ask if something is not clear):

1. Annotate the service Deployment with consul.hashicorp.com/connect-inject: "true", consul.hashicorp.com/connect-service: "<servicename>", and consul.hashicorp.com/connect-service-port: "<portnumber>". In my case that's necessary because I have injection disabled by default.
2. Annotate the ingress controller Deployment with consul.hashicorp.com/connect-inject: "true" and also with consul.hashicorp.com/connect-service-upstreams: "<servicename>:<localproxyport>". Let's assume localproxyport is 12345.
3. This is the critical part. Create an ExternalName service pointing to 127.0.0.1 in the same namespace as your Ingress, e.g.:

   apiVersion: v1
   kind: Service
   metadata:
   name: localhost
   namespace: <namespace>
   spec:
   externalName: 127.0.0.1
   type: ExternalName

4. Modify your Ingress and point it to the new localhost service, on `, e.g.:

   kind: Ingress
   ...
   spec:
   rules:
   - host: <hostname>
     http:
       paths:
       - backend:
           serviceName: localhost
           servicePort: <localproxyport>
         path: /
   ...

The key here is that localhost in the Ingress definition is not really your regular localhost, but rather localhost.<namespace>.svc.cluster.local, which resolves to 127.0.0.1 anyway, but it allows us to use it in the Ingress as a service name.

Let me know if this works!

[lkysow]

Looks like there might be a solution but we need to test this and document it.

[harindaka]

Is there any official documentation which outlines how to integrate and nginx based ingress with Consul and k8s services yet?

[lkysow]

Hi Helma, not yet. This is the ticket to watch though. When we have docs in about generic ingress we'll probably use nginx as an example.

[eikenb]

For anyone here interested, with release 0.23.0, consul-template now supports fetching the connect root and leaf certificates as well as the connect enable services. There are simple introductory documents describing how to create ingress gateways with NGINX and HAProxy as well.

[blake]

Consul 1.8 introduced support for a native Ingress gateway using Envoy. The gateway is configured using Consul's configuration entries. See this blog post on ingress gateways for an example of deploying on Kubernetes.

Later versions of Consul will allow gateways to be configured using CRDs.

[BlackRider97]

Is there any official documentation for Ingress to Service end-to-end mTLS communication ?

[lkysow]

Is there any official documentation for Ingress to Service end-to-end mTLS communication ?

Is this what you're looking for: https://www.consul.io/docs/k8s/connect/ingress-gateways?

[isaac88]

Hello @BlackRider97 @lkysow Do you have plan to integrate Kong Ingress with Consul Connect ?

[lkysow]

Hello @BlackRider97 @lkysow Do you have plan to integrate Kong Ingress with Consul Connect ?

I'm not aware of us working on that right now.

[lkysow]

With Consul ingress gateways released in Consul 1.8.0 I think we can close this issue. See our docs here: https://www.consul.io/docs/k8s/connect/ingress-gateways

[whiskeysierra]

The consul ingress-gateway doesn't support TLS termination so there are still reasons to have a proper kubernetes ingress controller running something like HAproxy or Nginx in front. Apart from the trick above (https://github.com/hashicorp/consul-k8s/issues/21#issuecomment-443823996) is there no official support to make consul connect nicely integrate with any existing ingress controller?

[blake]

@whiskeysierra The Ambassador API gateway is integrated with Consul. You can find a brief installation guide at https://www.consul.io/docs/k8s/connect/ambassador with more detailed docs available at https://www.getambassador.io/docs/latest/howtos/consul/.

There is also an active effort to integrate Traefik with Consul https://github.com/traefik/traefik/pull/7407 which I hope to see available soon.

[whiskeysierra]

I was hoping for a more generic way, so instead of needing individual ingress controllers start supporting individual service mesh providers, I'm looking for a way that uses the tools that are available to me today to do this. Ideally I'd want to use https://haproxy-ingress.github.io/, but I don't see how exactly I could make it talk to consul connect properly.

If I'm understanding the Ambassador connect integration correctly, it does not use consul ingress-gateway but is doing the mTLS connection to the target consul service (in our case kubernetes service + pods) directly. Is that correct?

[ishustava]

Hey @whiskeysierra, just chiming in this conversation.

I don't think there's a generic way to use any ingress gateway out-of-the-box with Consul. The main reason for this is that Consul has one Certificate Authority responsible for issuing certificates to services and its own authorization model that's responsible for deciding who's allowed to ask for those certificates. Unless you want to circumvent that security, the specific ingress providers would need to integrate with Consul directly.

Istio supports PERMISSIVE mode where your service will accept both plain text and mTLS, in which case using any ingress controller will work. However, Consul does not support an equivalent of a permissive mode, and so all connections to the backend services need to be via mTLS, which is what the Consul ingress gateway supports.

Hope this helps!

Iryna & @ndhanushkodi

[whiskeysierra]

What about this trick that was posted here before that runs the consul agent as a sidecar and routes all traffic to localhost. It bends kubernetes services/ingress concepts a bit but should work.

Alternatively, couldn't consul provide something that provides local DNS resolution for the ingress controller and resolves all kubernetes service DNS records to localhost? Again with a sidecar running there.

Both of these ways should a) be generic for any ingress controller because they use a proper consul agent, b) secure, because the only unencrypted traffic is locally within the pod.

[whiskeysierra]

The official docs that I found are:

1. https://www.consul.io/docs/k8s/connect/ingress-gateways
   Insecure, because it doesn't support TLS termination.
2. https://www.hashicorp.com/blog/proxy-ingress-to-consul-service-mesh
   Insecure because it doesn't do any hostname verification. It's also somewhat problematic to me since it tries to re-implement the functionality of a proper consul-agent sidecar by offloading the mTL to HAproxy and the certificate renewal/retrieval to consul-template.
3. https://www.haproxy.com/blog/haproxy-and-consul-with-dns-for-service-discovery/
   Seems like a proper way to do what option 2 was trying but uses the local consul agent as a nameserver (pretty much what I had in mind in my comment above).

[whiskeysierra]

In order to organize my thoughts on this, I tried to put all options in a table. Here it goes:

Option	K8s Ingress	K8s Ingress Controller	TLS Termination	Routing	Load Balancing	mTLS (ingress to service)	Certificates	Reference/Comment
A	n/a	n/a	n/a	consul	consul	consul	consul	https://www.consul.io/docs/k8s/connect/ingress-gateways
B	n/a	n/a	haproxy	haproxy	haproxy	haproxy1	consul-template2	https://www.hashicorp.com/blog/proxy-ingress-to-consul-service-mesh
C	Standard	haproxy-ingress	haproxy	haproxy	kubernetes	haproxy3	consul-template2	Option B, but delegate load balancing to K8s
D	ExternalName "localhost"	haproxy-ingress	haproxy	haproxy	consul	consul	consul	https://github.com/hashicorp/consul-k8s/issues/21#issuecomment-443823996
E	ExternalName + Consul DNS	haproxy-ingress	haproxy	haproxy	consul	consul	consul	https://www.haproxy.com/blog/haproxy-and-consul-with-dns-for-service-discovery/

¹ Missing hostname verification because verifyhost is not specified ² A custom sidecar using consul-template would be needed to continuously retrieve/renew client and CA certificates ³ Support for verifyhost needs to be added to haproxy-ingress (not really relevant here)

Option A is the only one that uses a Consul Ingress Gateway but because it lacks TLS termination and Kubernetes Ingress support, i.e. it's not usable.
Option B and C both require a custom sidecar that is doing the certificate management, i.e. additional development effort.
Option D technically works, but (as mentioned in an earlier comment) feels really weird to have a fake localhost kubernetes service of type ExternalName. Option E feels like the cleanest solution to me so far. It also uses ExternalName services, but makes use of Consul DNS to have proper hostnames for services.

I've use haproxy and haproxy-ingress as examples above. One could easily generalize it and make this work for any ingress controller, assuming they support the following:

• the consul agent can be attached as a sidecar
• the ingress controller supports custom name servers and can be configured to use 127.0.0.1:8600

Benefits, compared to the ExternalName "localhost" trick (see https://github.com/hashicorp/consul-k8s/issues/21#issuecomment-443823996), are:

• uses proper hostname instead of localhost for an external name service (rather cosmetic, but still)
• no changes to the Deployment for each new ingress/route (needed a port mapping before)
• consul-agent is used in a normal way, no magic

Sample ingress using Option E:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-host-ingress
  namespace: ingress
  annotations:
    ingress.class: haproxy
spec:
  rules:
    - host: my-host.example.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-service
                port:
                  name: http
      tls:
        secretName: my-host-tls
        hosts:
          - my-host.example.org
---
apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: ingress
spec:
  externalName: my-service.service.consul
  type: ExternalName
  ports:
    - name: http
      protocol: TCP
      port: 8080
      targetPort: 8080

[whiskeysierra]

I just realized that Option E would resolve the Pod IPs and not to localhost. So it wouldn't work.

[ishustava]

Hey @whiskeysierra

https://www.consul.io/docs/k8s/connect/ingress-gateways Insecure, because it doesn't support TLS termination.

You can configure Consul Ingress gateways with a TLS listener. Sorry that I missed this concern in your original comment.

What about this trick that was posted here before that runs the consul agent as a sidecar and routes all traffic to localhost

Yeah, you can use that trick, however, it certainly seems like a workaround, especially now that Consul has Ingress Gateway support. Two things to keep in mind with that workaround:

1. You will be adding an extra proxy (envoy sidecar) alongside your ingress.
2. You will need to re-deploy your ingress gateway anytime you need to add a new service (upstream) that the ingress needs to forward connections to.

Option A is the only one that uses a Consul Ingress Gateway but because it lacks TLS termination and Kubernetes Ingress support, i.e. it's not usable.

We do have support for an IngressGateway CR though so you can just configure everything with yaml. Not sure if this helps your case.

[whiskeysierra]

Does the consul ingress gateway support different certificates for different domains? We also need client certificate validation.

[ndhanushkodi]

Hey @whiskeysierra!

Does the consul ingress gateway support different certificates for different domains?

Currently there is only one certificate, where each host defined in the Host field will be added as a DNSSAN to the gateway's x509 certificate.

We also need client certificate validation.

I see no indication in the ingress implementation that we currently support client certificate validation, if you would like to see that feature requested, would you mind opening that as a separate issue (feature-request) to support client certificate validation in the ingress gateway? I think it would be appropriate to open that in https://github.com/hashicorp/consul and then we would add support on the K8s side as well.

[whiskeysierra]

I think I'd rather invest in getting consul connect to nicely integrate with any kubernetes ingress controller without reinventing the wheel.

[ndhanushkodi]

@whiskeysierra Yes that's a valid point, can you open a separate feature request for ingress controller support with Consul Connect? That way we can independently track community interest in that (rather than in this closed issue), since this specific issue seemed to be resolved earlier using Consul Ingress Gateway support. The new issue will help us prioritize supporting your ask of integrating consul connect with k8s ingress.

Supporting generic ingress controllers would require some investigation into the Options B, C, and D in your table and exploring other options to support it as well.

[whiskeysierra]

@ndhanushkodi You're absolutely right. I opened https://github.com/hashicorp/consul-k8s/issues/430.

[pedrohdz]

For those using the ExternalName trick from https://github.com/hashicorp/consul-k8s/issues/21#issuecomment-443823996 with ingress-nginx. Set the externalName to localhost. It would looks something like:

apiVersion: v1
kind: Service
metadata:
  name: localhost
  namespace: <namespace>
spec:
  externalName: localhost
  type: ExternalName

The documentation for ServiceSpec states:

externalName is the external reference that discovery mechanisms will return as an alias for this service (e.g. a DNS CNAME record). No proxying will be involved.

Using a externalName of 127.0.0.1 results in the following error with ingress-nginx:

2021/02/25 13:51:16 [error] 37#37: *4835 [lua] dns.lua:152: dns_lookup(): failed to query the DNS server for 127.0.0.1:

[pedrohdz]

Some more notes on the solutions outlined in https://github.com/hashicorp/consul-k8s/issues/21#issuecomment-443823996 and https://github.com/hashicorp/consul-k8s/issues/21#issuecomment-769726419.

The Service name can be anything, not limited to localhost which makes it a little less confusing, in my mind at least. :-) Would help in the instances when you want to define more than one Consul connected service, lets you have multiple Service resources.

In the end, I ended up with something like:

---
apiVersion: v1
kind: Service
metadata:
  name: local-static-server
  namespace: ingress-to-consul
spec:
  externalName: localhost
  type: ExternalName
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 1234

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: consul-internal-ingress
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: local-static-server
  namespace: ingress-to-consul
spec:
  rules:
    - host: consul-demo.demo.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: local-static-server
                port:
                  name: http
  tls:
    - hosts:
        - consul-demo.demo.com
      secretName: demo-local-wildcard-tls-certs
...