Support multiple services and sidecars per pod

[vinay-g]

Consul connect currently supports ( according to documentation) only exposing one port per pod like below:

consul.hashicorp.com/connect-service consul.hashicorp.com/connect-service-port

Is there a way to provide multiple ports, possibly with different service names.

[banks]

Hey, great question.

In general no that is not supported currently.

Connect's internal model is that each sidecar proxy corresponds to exactly one service instance. If your process exposes two separate APIs on separate ports then although it's one process it's logically two separate services in Consul's catalog.

We don't have plans to have a single sidecar proxy process be configured as two separate services for now. That might be a feature request for Consul in general if this setup is common enough and the overhead of multiple Envoys per pod is a real issue.

You could run two separate Envoy sidecars in the pod one for each "service" exposed and all would work fine. I've changed the issue to reflect that it's a possible future feature the injector could support without changes to Consul.

You could still do it manually in the interim by explicitly adding Envoy containers and necessary bootstrap config/registration to the pod spec. If you look at the actual pod Yaml from kube API after registering with an injected Envoy you can see what the injector added to help show what you'd need to add manually.

I'd be interested to know how many other people benefit from this setup to decide if it's worth allowing the injector to do that in general in the future. I can imagine there are several reasons to expose several ports/services from a pod.

[vinay-g]

Thanks for the suggestion, we can try that.

[vinay-g]

Hi I have a follow up question not sure if this the right place. Is it possible to achieve the above functionality of multiple service per pod using a single sidecar ? Manual steps are good enough for us.

[banks]

Is it possible to achieve the above functionality of multiple service per pod using a single sidecar ?

I might be possible using the "escape hatch" documented here: https://www.consul.io/docs/connect/proxies/envoy.html#advanced-listener-configuration

To configure that you'd also need to bypass the kube injector as there is no way to inject that config into the proxy registration currently (will be soonish).

In that case you are basically overriding Consul Connect's management of the proxy config and have to write your own Envoy configs by hand which is not especially pleasant.

The other issue you'll hit is that we still only set up one public listener/port which means you'll only be able to serve multiple services if they have L7 support in Envoy (e.g. HTTP) so you can determine which incoming traffic should go to which service by hostname or path or similar. That will depend on how you name services and address request too!

So it's just about possible I'd imagine assuming HTTP services, but not really pleasant. It recommend multiple Envoys for now and we'll consider how we could potentially allow one single Envoy to register as two services.

The mechanics of configuring one envoy to serve two services are not really hard - we could make that change in Consul without that much code change. A more subtle problem is the security posture - currently each Envoy authenticates to Consul with a token that should only have permissions for the service its registering as. We could make envoy present multiple tokens, or could make it work only if the token has permissions for both target services but would take some thought. And then for outbound requests we'd need a way of knowing which service's certificate to use as the client one. What would happen if the two services defined upstream listeners on the same local port etc.

This is not really the place for that though as it's a Consul change not a consul-k8s one - please feel free to open a feature request on Consul repo and we can consider when it makes sense to solve that problem. I feel like some of the things in the pipeline currently will maybe make that simpler to support.

[vinay-g]

Thank you for the detailed input, appreciate the help. I will create an enhancement on consul and link it here if other have similar requests.

https://github.com/hashicorp/consul/issues/5388

[lkysow]

I'm going to close this here because it is being tracked in that Consul issue.