Allow serviceAddress to accept dns record

Tahvok commented 2 years ago

Description

This is for kuma universal. When using kuma-dp inside a user-defined docker network bridge, each container listens on its own ip. Therefore, you must set serviceAddress inside inbound, like so:

type: Dataplane
mesh: default
name: {{ name }}
networking:
  address: kuma-dp
  advertisedAddress: {{ host_address }}
  inbound:
    - port: 1337
      servicePort: 80
      serviceAddress: 172.18.0.5
      tags:
        kuma.io/service: my-service

However, the service container ip might change, for example recreation of a container will assign a new ip to the new container (even if same name is used).

This leads to an issue where each time the service container changes, you need to adjust the kuma-dp serviceAddress to the new address.

However, docker itself provides a solution, where you can communicate directly with each service inside same docker network bridge, by exposing a special dns service, where each container can communicate with each other using the container name. You can read more about it here and here.

This feature request is asking to add the capability to allow setting serviceAddress to a dns name and not only ip address. This will allow kuma to discover the ip address of the service container name, and the configuration of kuma-dp can be greatly simplified.

DNS discovery is partly supported already. kuma-dp is allowing to set networking.address to be DNS name, and essentially you can set it to be the name of the container kuma-dp as in this example. This work was done in pr #965 . It appears to me that although networking.address does accept a dns record, and it does not fail to startup, it does not able to load the listener (no errors are thrown). I could not manage to debug why that is happening.

In the end, I want to be able to set the configuration as follows:

type: Dataplane
mesh: default
name: {{ name }}
networking:
  address: kuma-dp
  advertisedAddress: {{ host_address }}
  inbound:
    - port: 1337
      servicePort: 80
      serviceAddress: my-service
      tags:
        kuma.io/service: service-name

jakubdyszkiewicz commented 2 years ago

Hey,

could you send a bit more details about the topology of this deployment? Do you run many services behind one kuma-dp container or is it 1:1 kuma-dp:service? How do you manage Dataplane object? Is it direct or indirect https://kuma.io/docs/1.8.x/explore/dpp-on-universal/#lifecycle ? Do you use docker-compose to deploy those containers?

Tahvok commented 2 years ago

I'll try to explain in words. Each instance (or machine or host) is running only 1 service. It's a container that is part of a docker network bridge, it can have some additional sidecar, which is why docker bridge is used, for seamless communication between the service and the sidecars. Inside same docker network bridge we bring up kuma-dp, essentially as another sidecar. This also answers your second question - we are running 1:1 kuma-dp:service - each service has a kuma-dp of its own.

This is very similar to how pods are working in kubernetes, except that each container has ip of its own, and they are not sharing the same interface.

We manage each dataplane using direct mode. Basically we run a deploy tool that brings up the instance(s) and deploys the service container and its sidecar.

Regarding your last question, docker compose brings up a regular docker network bridge for all containers in the same compose file - we are not using docker compose, and instead using direct docker commands, but I expect the same issue to happen with docker compose. It is also expected from docker compose that you use container names for communication between containers in the same docker network: https://docs.docker.com/compose/networking/

jakubdyszkiewicz commented 1 year ago

Triage:

Thanks for the context.

In order to implement this, we would need to change the type of the inbound cluster from EDS to STRICT_DNS and skip delivering inbound endpoints via EDS. This way it's Envoy that can resolve the DNS, not the CP. This can only happen when serviceAddress is not IP, we don't want to default to STRICT_DNS.

At this moment, our team has other priorities, but we are happy to help with the implementation if you are interested to contribute. You can always reach out to us on Kuma slack.