search

hashicorp / consul-k8s

First-class support for Consul Service Mesh on Kubernetes
https://www.consul.io/docs/k8s
Mozilla Public License 2.0
669 stars 322 forks source link

Get proper clusterIP on service type: ClusterIP #99

Open daniego opened 5 years ago

daniego commented 5 years ago

Hi, I'm not sure if I missed something in my configuration but looks like that when a service is "ClusterIP" the sync-catalog synchronises the endpoint addresses rather than the actual clusterIP. Is that by design or there is a way to get the cluster IP and the ports?

pmatv commented 5 years ago

Hi, I'm seeing the same behavior as @daniego. Here is an example:

$ kubectl describe svc staging-consul-dns -n consul
Name:              staging-consul-dns
Namespace:         consul
Labels:            app=consul
                   chart=consul-helm
                   heritage=Tiller
                   release=staging
Annotations:       <none>
Selector:          app=consul,hasDNS=true,release=staging
Type:              ClusterIP
IP:                100.69.141.243
Port:              dns-tcp  53/TCP
TargetPort:        dns-tcp/TCP
Endpoints:         100.106.229.106:8600,100.114.45.206:8600,100.114.45.208:8600 + 6 more...
Port:              dns-udp  53/UDP
TargetPort:        dns-udp/UDP
Endpoints:         100.106.229.106:8600,100.114.45.206:8600,100.114.45.208:8600 + 6 more...
Session Affinity:  None
Events:            <none>

# curl localhost:8500/v1/catalog/service/staging-consul-dns
[
    {
        "ID": "",
        "Node": "k8s-sync",
        "Address": "127.0.0.1",
        "Datacenter": "dc-staging",
        "TaggedAddresses": null,
        "NodeMeta": {
            "external-source": "kubernetes"
        },
        "ServiceKind": "",
        "ServiceID": "staging-consul-dns-4968a295cbbb",
        "ServiceName": "staging-consul-dns",
        "ServiceTags": [
            "k8s"
        ],
        "ServiceAddress": "100.116.16.124",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {
            "external-k8s-ns": "",
            "external-source": "kubernetes",
            "port-dns-tcp": "53",
            "port-dns-udp": "53"
        },
        "ServicePort": 53,
        "ServiceEnableTagOverride": false,
        "ServiceProxyDestination": "",
        "ServiceProxy": {},
        "ServiceConnect": {},
        "CreateIndex": 17130,
        "ModifyIndex": 17130
    },
    {
        "ID": "",
        "Node": "k8s-sync",
        "Address": "127.0.0.1",
        "Datacenter": "dc-staging",
        "TaggedAddresses": null,
        "NodeMeta": {
            "external-source": "kubernetes"
        },
        "ServiceKind": "",
        "ServiceID": "staging-consul-dns-857593e28e5c",
        "ServiceName": "staging-consul-dns",
        "ServiceTags": [
            "k8s"
        ],
        "ServiceAddress": "100.106.229.106",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {
            "external-k8s-ns": "",
            "external-source": "kubernetes",
            "port-dns-tcp": "53",
            "port-dns-udp": "53"
        },
        "ServicePort": 53,
        "ServiceEnableTagOverride": false,
        "ServiceProxyDestination": "",
        "ServiceProxy": {},
        "ServiceConnect": {},
        "CreateIndex": 17117,
        "ModifyIndex": 17117
    },
    {
        "ID": "",
        "Node": "k8s-sync",
        "Address": "127.0.0.1",
        "Datacenter": "dc-staging",
        "TaggedAddresses": null,
        "NodeMeta": {
            "external-source": "kubernetes"
        },
        "ServiceKind": "",
        "ServiceID": "staging-consul-dns-8917e803fbd6",
        "ServiceName": "staging-consul-dns",
        "ServiceTags": [
            "k8s"
        ],
        "ServiceAddress": "100.116.16.112",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {
            "external-k8s-ns": "",
            "external-source": "kubernetes",
            "port-dns-tcp": "53",
            "port-dns-udp": "53"
        },
        "ServicePort": 53,
        "ServiceEnableTagOverride": false,
        "ServiceProxyDestination": "",
        "ServiceProxy": {},
        "ServiceConnect": {},
        "CreateIndex": 17161,
        "ModifyIndex": 17161
    },
    {
        "ID": "",
        "Node": "k8s-sync",
        "Address": "127.0.0.1",
        "Datacenter": "dc-staging",
        "TaggedAddresses": null,
        "NodeMeta": {
            "external-source": "kubernetes"
        },
        "ServiceKind": "",
        "ServiceID": "staging-consul-dns-92474e5d2418",
        "ServiceName": "staging-consul-dns",
        "ServiceTags": [
            "k8s"
        ],
        "ServiceAddress": "100.114.45.208",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {
            "external-k8s-ns": "",
            "external-source": "kubernetes",
            "port-dns-tcp": "53",
            "port-dns-udp": "53"
        },
        "ServicePort": 53,
        "ServiceEnableTagOverride": false,
        "ServiceProxyDestination": "",
        "ServiceProxy": {},
        "ServiceConnect": {},
        "CreateIndex": 17160,
        "ModifyIndex": 17160
    },
    {
        "ID": "",
        "Node": "k8s-sync",
        "Address": "127.0.0.1",
        "Datacenter": "dc-staging",
        "TaggedAddresses": null,
        "NodeMeta": {
            "external-source": "kubernetes"
        },
        "ServiceKind": "",
        "ServiceID": "staging-consul-dns-b2d8e263de0c",
        "ServiceName": "staging-consul-dns",
        "ServiceTags": [
            "k8s"
        ],
        "ServiceAddress": "100.99.205.249",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {
            "external-k8s-ns": "",
            "external-source": "kubernetes",
            "port-dns-tcp": "53",
            "port-dns-udp": "53"
        },
        "ServicePort": 53,
        "ServiceEnableTagOverride": false,
        "ServiceProxyDestination": "",
        "ServiceProxy": {},
        "ServiceConnect": {},
        "CreateIndex": 17125,
        "ModifyIndex": 17125
    },
    {
        "ID": "",
        "Node": "k8s-sync",
        "Address": "127.0.0.1",
        "Datacenter": "dc-staging",
        "TaggedAddresses": null,
        "NodeMeta": {
            "external-source": "kubernetes"
        },
        "ServiceKind": "",
        "ServiceID": "staging-consul-dns-cd67fdb76396",
        "ServiceName": "staging-consul-dns",
        "ServiceTags": [
            "k8s"
        ],
        "ServiceAddress": "100.120.187.109",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {
            "external-k8s-ns": "",
            "external-source": "kubernetes",
            "port-dns-tcp": "53",
            "port-dns-udp": "53"
        },
        "ServicePort": 53,
        "ServiceEnableTagOverride": false,
        "ServiceProxyDestination": "",
        "ServiceProxy": {},
        "ServiceConnect": {},
        "CreateIndex": 17131,
        "ModifyIndex": 17131
    },
    {
        "ID": "",
        "Node": "k8s-sync",
        "Address": "127.0.0.1",
        "Datacenter": "dc-staging",
        "TaggedAddresses": null,
        "NodeMeta": {
            "external-source": "kubernetes"
        },
        "ServiceKind": "",
        "ServiceID": "staging-consul-dns-d893d20ed955",
        "ServiceName": "staging-consul-dns",
        "ServiceTags": [
            "k8s"
        ],
        "ServiceAddress": "100.114.45.206",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {
            "external-k8s-ns": "",
            "external-source": "kubernetes",
            "port-dns-tcp": "53",
            "port-dns-udp": "53"
        },
        "ServicePort": 53,
        "ServiceEnableTagOverride": false,
        "ServiceProxyDestination": "",
        "ServiceProxy": {},
        "ServiceConnect": {},
        "CreateIndex": 17164,
        "ModifyIndex": 17164
    },
    {
        "ID": "",
        "Node": "k8s-sync",
        "Address": "127.0.0.1",
        "Datacenter": "dc-staging",
        "TaggedAddresses": null,
        "NodeMeta": {
            "external-source": "kubernetes"
        },
        "ServiceKind": "",
        "ServiceID": "staging-consul-dns-e672f0c6ce5c",
        "ServiceName": "staging-consul-dns",
        "ServiceTags": [
            "k8s"
        ],
        "ServiceAddress": "100.117.99.215",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {
            "external-k8s-ns": "",
            "external-source": "kubernetes",
            "port-dns-tcp": "53",
            "port-dns-udp": "53"
        },
        "ServicePort": 53,
        "ServiceEnableTagOverride": false,
        "ServiceProxyDestination": "",
        "ServiceProxy": {},
        "ServiceConnect": {},
        "CreateIndex": 17165,
        "ModifyIndex": 17165
    },
    {
        "ID": "",
        "Node": "k8s-sync",
        "Address": "127.0.0.1",
        "Datacenter": "dc-staging",
        "TaggedAddresses": null,
        "NodeMeta": {
            "external-source": "kubernetes"
        },
        "ServiceKind": "",
        "ServiceID": "staging-consul-dns-f24c4cb3d420",
        "ServiceName": "staging-consul-dns",
        "ServiceTags": [
            "k8s"
        ],
        "ServiceAddress": "100.120.187.111",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {
            "external-k8s-ns": "",
            "external-source": "kubernetes",
            "port-dns-tcp": "53",
            "port-dns-udp": "53"
        },
        "ServicePort": 53,
        "ServiceEnableTagOverride": false,
        "ServiceProxyDestination": "",
        "ServiceProxy": {},
        "ServiceConnect": {},
        "CreateIndex": 17120,
        "ModifyIndex": 17120
    }
]
daniego commented 5 years ago

Hi, any update on this issue?

lkysow commented 5 years ago

Hi Daniel, this is by design. We could make it configurable though.

Can you elaborate on your use-case as to why you want the clusterip registered rather than the individual endpoints?

praymann commented 5 years ago

DNS in Daniel's case, and statsd in my situation, are probably bad examples but generally I think it's safe to say that by sync'in endpoint addresses we lose out on all the affinity/balancing that the service ip address would do through kube-proxy and instead have to figure that out on the client side when we get 4 addresses back from Consul instead of just one.

It is also the case that by doing endpoint addresses it breaks the idea that a ClusterIP's Service's IP is fairly static and can be trusted to not disappear whereas the pods behind it can be ephemeral and disappear often. Why should discovering through Consul be different than the built-in records of my-svc.my-namespace.svc.cluster-domain.example which return the single ClusterIP Service Address not all the pod addresses?

Perhaps introducing a new flag of --sync-clusterip-services-as-endpoints with a default of true, and handling the logic when the flag is false here https://github.com/hashicorp/consul-k8s/blob/master/catalog/from-k8s/resource.go#L480 is the easiest and best approach to handle both use cases?

praymann commented 5 years ago

I like the idea of doing something like:

    // annotationServiceAsEndpoints is valid only on ClusterIP services.
    // It specifics whether or not to sync the service with each individual endpoint or
    // as the singular IP of the service. If this isn't set then the
    // default based on the syncer configuration is chosen.
    annotationServiceAsEndpoints = "consul.hashicorp.com/service-as-endpoints"

as well.

But it looks like there is some of the resource.go that tracks and watches endpoints which would need to be handled as well. Not sure if it's best to just by default track the IP and the endpoints or try to separate them.

lkysow commented 5 years ago

Thanks for letting us know your use-case!

For background, the reason we sync Pod IPs instead of the ClusterIP is because we assumed the consumers of the synced services would not be running in Kubernetes and so wouldn't be able to use the Cluster IP (whereas Pod IPs might be routable in some clouds). This assumption is incorrect in your case 😄.

daniego commented 4 years ago

In my scenario I'm templating an HaProxy instance and need to use the cluster IP rather than the actual pods IPs to preserve the real client IP. Using the service DNS like my-svc.my-namespace.svc.cluster-domain.example partially works. When haproxy starts it resolves the DNS and keep using the resolved IPs. If the service gets recreated the IP will change and HaProxy is not longer able to reach it.

daniego commented 4 years ago

There is any updated on this issue?

vladimir-avinkin commented 4 years ago

I think that @praymann described the use case very well. I would also like to see this implemented, in many scenarios service ips are also routable, same as pod ips. In that case it's preferable to use them as service endpoints for the same reasons services are implemented in k8s in the first place, to serve as persistent virtual ip l4 loadbalancers