Headless services not working

[draese]

Describe the bug I try to reference a headless service. In my example, I have a deployment of three Kafka brokers (kafka-test-1, kafka-test-2 and kafka-test-3) being behind the headless service kafka-test. Once the headless service is resolved, its FQDN with the K8s cluster, its DNS entry is kafka-test-1.kafka-test.kafka.svc.cluster.local (kafka being the namespace)

This service cannot be created on my local machine (added to /etc/hosts) also not via KubernetesLocalProcessConfig.yaml. If I try to add the FQDN for services in the yaml file, I get an error about the service name format.

To Reproduce Here the definition of my headless service for reference:

apiVersion: v1 kind: Service metadata: annotations: meta.helm.sh/release-name: kafka-test meta.helm.sh/release-namespace: kafka creationTimestamp: "2021-03-26T01:11:04Z" labels: app.kubernetes.io/component: kafka app.kubernetes.io/instance: kafka-test app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: kafka helm.sh/chart: kafka-12.13.2 managedFields:

• apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:meta.helm.sh/release-name: {} f:meta.helm.sh/release-namespace: {} f:labels: .: {} f:app.kubernetes.io/component: {} f:app.kubernetes.io/instance: {} f:app.kubernetes.io/managed-by: {} f:app.kubernetes.io/name: {} f:helm.sh/chart: {} f:spec: f:clusterIP: {} f:ports: .: {} k:{"port":9092,"protocol":"TCP"}: .: {} f:name: {} f:port: {} f:protocol: {} f:targetPort: {} k:{"port":9093,"protocol":"TCP"}: .: {} f:name: {} f:port: {} f:protocol: {} f:targetPort: {} f:selector: .: {} f:app.kubernetes.io/component: {} f:app.kubernetes.io/instance: {} f:app.kubernetes.io/name: {} f:sessionAffinity: {} f:type: {} manager: Go-http-client operation: Update time: "2021-03-26T01:11:04Z" name: kafka-test-headless namespace: kafka resourceVersion: "688603" selfLink: /api/v1/namespaces/kafka/services/kafka-test-headless uid: 5d0ed046-6a95-4bfc-a6de-e02ed1b7f890 spec: clusterIP: None ports:
• name: tcp-client port: 9092 protocol: TCP targetPort: kafka-client
• name: tcp-internal port: 9093 protocol: TCP targetPort: kafka-internal selector: app.kubernetes.io/component: kafka app.kubernetes.io/instance: kafka-test app.kubernetes.io/name: kafka sessionAffinity: None type: ClusterIP status: loadBalancer: {}

[pragyamehta]

Hi @draese Thanks for opening the issue! We just released the support for headless services this morning on VSCode for non-isolated scenarios. Please make sure you have the latest version of the VSCode extension installed and try to debug again (no additional changes needed) and let us know if you face any issues.

[draese]

Thanks for the quick response! I have the version v1.0.120210401 installed, which seems to be the latest that I can use right now?

The headless services are actually resolved: Service 'kafka-test-headless.kafka' is available on 127.1.1.4:9092. Service 'kafka-test-headless.kafka' is available on 127.1.1.4:9093.

You see here (on my Windows dev env) that the headless kafka service is added twice, using different ports.

When running the Kafka client against this (configured to connect to kafka-test-headless.kafka.svc.cluster.local), I get the following error messages from the Kafka client:

java.net.UnknownHostException: kafka-test-0.kafka-test-headless.kafka.svc.cluster.local at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[na:na] at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509) ~[na:na] at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368) ~[na:na] at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302) ~[na:na] at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) ~[kafka-clients-2.7.0.jar:na] at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:501) ~[kafka-clients-2.7.0.jar:na] at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:458) ~[kafka-clients-2.7.0.jar:na] at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:169) ~[kafka-clients-2.7.0.jar:na] at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1002) ~[kafka-clients-2.7.0.jar:na] at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:75) ~[kafka-clients-2.7.0.jar:na] at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1175) ~[kafka-clients-2.7.0.jar:na] at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1063) ~[kafka-clients-2.7.0.jar:na] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:559) ~[kafka-clients-2.7.0.jar:na] at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:325) ~[kafka-clients-2.7.0.jar:na] at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:240) ~[kafka-clients-2.7.0.jar:na] at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

The service was defined in KubernesLocalProcessConfig.yaml as:

• name: KAFKA_HEADLESS value: $(services:kafka-test-headless.kafka)

The problem seems to be that the actual brokers here are (kafka-test-[1..3] are not created in the .,,/ets/hosts an therefore not resolvable. The Kafka client receives the list of brokers from the first broker it connects to.

The real problem is, that I can't add these three (kafka-test-[1..3]) services to the KubernetesLocalProcessConfig file. If I try something lilke

• name: KAFKA_HEADLESS1 value: $(services:kafka-test-1.kafka-test-headless.kafka)

I get an error as the FQDN for the broker is not supported by BridgeToK8s...

[amsoedal]

@draese I believe I identified the issue. Fix should be out with our release later this month, I'll let you know when it's available!

[draese]

Thanks!

[bruno-brant]

I'm having the same problem as the OP. This problem seems unrelated to headless services, instead, something specific to Kafka - when a client connects to a Kafka broker, it receives a hostname for the kafka service and is directed to connect to it. If the Bridge doesn't create the necessary entries in the machine hosts files, the client won't be able to find those services.

[amsoedal]

@bruno-brant are you using VS Code? The issue should be fixed there