[bitnami/kafka] 14.2.0 cannot connect to brokers from the outside using the cluster's default service when type is NodePort

[jmastr]

Which chart: bitnami/kafka-14.2.0

Describe the bug I use this configuration:

externalAccess.enabled=true
externalAccess.service.type=LoadBalancer
externalAccess.autoDiscovery.enabled=true
serviceAccount.create=true
rbac.create=true

which works beautifully. I get:

% kubectl --namespace kafka get svc                                                                                                                               
NAME                       TYPE           CLUSTER-IP       EXTERNAL-IP                                                                        PORT(S)                         AGE
kafka                      LoadBalancer   10.100.168.158   a3bb452df05a148c78ea161b8469e5eb-684a52fdc7d8af53.elb.eu-central-1.amazonaws.com   9092:30993/TCP,9094:31471/TCP   16d
kafka-0-external           LoadBalancer   10.100.64.36     a6fb836e9d3134a4faef768338545265-24dbaf79d87bcbaf.elb.eu-central-1.amazonaws.com   9094:32110/TCP                  20h
...

I can consume a topic that I created:

% kafkacat -C -b a3bb452df05a148c78ea161b8469e5eb-684a52fdc7d8af53.elb.eu-central-1.amazonaws.com:9094 -X security.protocol=SASL_PLAINTEXT -X sasl.mechanism=SCRAM-SHA-512 -X sasl.username=user -X sasl.password=PASSWORD -t test-03
foo
bar
% Reached end of topic test-03 [0] at offset 2

Switching externalAccess.service.type to NodePort with:

externalAccess.enabled=true
externalAccess.service.type=NodePort
externalAccess.autoDiscovery.enabled=true
serviceAccount.create=true
rbac.create=true

I get:

% kubectl --namespace kafka get svc                                                                                                                               
NAME                       TYPE           CLUSTER-IP       EXTERNAL-IP                                                                        PORT(S)                         AGE
kafka                      LoadBalancer   10.100.168.158   a3bb452df05a148c78ea161b8469e5eb-684a52fdc7d8af53.elb.eu-central-1.amazonaws.com   9092:30993/TCP,9094:31471/TCP   16d
kafka-0-external           NodePort       10.100.64.36     <none>                                                                             9094:32110/TCP                  20h
...

and consumption just hangs:

% kafkacat -C -b a3bb452df05a148c78ea161b8469e5eb-684a52fdc7d8af53.elb.eu-central-1.amazonaws.com:9094 -X security.protocol=SASL_PLAINTEXT -X sasl.mechanism=SCRAM-SHA-512 -X sasl.username=user -X sasl.password=PASSWORD -t test-03
^C

The README states:

Following the aforementioned steps will also allow to connect the brokers from the outside using the cluster's default service (when service.type is LoadBalancer or NodePort). Use the property service.externalPort to specify the port used for external connections.

To Reproduce Steps to reproduce the behavior:

Use above configuration and switch externalAccess.service.type from LoadBalancer to NodePort.

Expected behavior Being able to consume topics via externally exposed default service when externalAccess.service.type is NodePort.

Version of Helm and Kubernetes:

• Output of helm version:

version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"clean", GoVersion:"go1.13.12"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.7-eks-d88609", GitCommit:"d886092805d5cc3a47ed5cf0c43de38ce442dfcb", GitTreeState:"clean", BuildDate:"2021-07-31T00:29:12Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}

Additional context

[alemorcuq]

Hi!

$ kafkacat -C -b a3bb452df05a148c78ea161b8469e5eb-684a52fdc7d8af53.elb.eu-central-1.amazonaws.com:9094 -X security.protocol=SASL_PLAINTEXT -X sasl.mechanism=SCRAM-SHA-512 -X sasl.username=user -X sasl.password=PASSWORD -t test-03

It looks like you are trying to connect to your LoadBalancer service instead to the NodePort service.

[jmastr]

Thanks for your response!

Exactly. I want to expose the default service (kafka) via a LoadBalancer to reach my three Kafka pods.

Therefore I enable the external services (kafka-0-external, kafka-1-external, and kafka-2-external). If I give these services type LoadBalancer I can access the default service from the outside. If I give them the type NodePort I cannot, although the docs state otherwise.

[juan131]

Hi @jmastr

I don't fully understand why you need to expose the default service (I mean using LoadBalancer type) if you're also exposing each Kafka broker individually.

Wouldn't be easier to keep this default service internal (ClusterIP type) and only use LoadBalancer for the external services? Then you'll have to configure your consumer/producer with the external IP of every broker.

Going back to the original issue, could you share the exact values you were using when you found the issue with exposing Kafka via NodePort?

[jmastr]

Hey @juan131

Wouldn't be easier to keep this default service internal (ClusterIP type) and only use LoadBalancer for the external services? Then you'll have to configure your consumer/producer with the external IP of every broker.

That was our first solution and we now rolled back to it. Being not very familiar with Kafka we wanted to see which other options we might have. Main purpose was to reduce n LoadBalancers (for every broker one LB) to only one LoadBalancer and let Kubernetes do the balancing.

Going back to the original issue, could you share the exact values you were using when you found the issue with exposing Kafka via NodePort?

Quite frankly I cannot share more values then I already did above. We could do a screenshare and I show the problem/behaviour?

[juan131]

Hi @jmastr

Quite frankly I cannot share more values then I already did above

I guess you're using this setup then, right?

replicaCount: 3
externalAccess:
  enabled: true
  service:
    type: NodePort
  autoDiscovery:
    enabled: true
serviceAccount:
  create: true
rbac:
  create: true

I installed the chart using these exact values and I found no issues, see:

$ k get pods
NAME                READY   STATUS    RESTARTS   AGE
kafka-0             1/1     Running   0          5m1s
kafka-1             1/1     Running   0          5m1s
kafka-2             1/1     Running   0          5m
kafka-zookeeper-0   1/1     Running   0          5m1s
$ k get svc
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
kafka                      ClusterIP   10.227.251.98    <none>        9092/TCP                     6m30s
kafka-0-external           NodePort    10.227.243.55    <none>        9094:31780/TCP               6m30s
kafka-1-external           NodePort    10.227.245.36    <none>        9094:32728/TCP               6m30s
kafka-2-external           NodePort    10.227.242.119   <none>        9094:30396/TCP               6m30s
kafka-headless             ClusterIP   None             <none>        9092/TCP,9093/TCP            6m30s
kafka-zookeeper            ClusterIP   10.227.247.214   <none>        2181/TCP,2888/TCP,3888/TCP   6m30s
kafka-zookeeper-headless   ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP   6m30s

I also tried to connect to the installation NOTES:

(...)
To connect to your Kafka server from outside the cluster, follow the instructions below:

    Kafka brokers domain: You can get the external node IP from the Kafka configuration file with the following commands (Check the EXTERNAL listener)

        1. Obtain the pod name:

        kubectl get pods --namespace default -l "app.kubernetes.io/name=kafka,app.kubernetes.io/instance=kafka,app.kubernetes.io/component=kafka"

        2. Obtain pod configuration:

        kubectl exec -it KAFKA_POD -- cat /opt/bitnami/kafka/config/server.properties | grep advertised.listeners

    Kafka brokers port: You will have a different node port for each Kafka broker. You can get the list of configured node ports using the command below:

        echo "$(kubectl get svc --namespace default -l "app.kubernetes.io/name=kafka,app.kubernetes.io/instance=kafka,app.kubernetes.io/component=kafka,pod" -o jsonpath='{.items[*].spec.ports[0].nodePort}' | tr ' ' '\n')"

This is how my Conduktor configuration looks like:

[jmastr]

Hey @juan131

Thanks for setting up your test environment.

I guess you're using this setup then, right?

Correct

Now set service.type to type LoadBalancer, if your setup allows, NodePort otherwise (this I did not test though) and try using that LB or NodePort directly instead of the dedicated broker's node ports.

For me that fails when using kafkacat. If externalAccess.service.type is LoadBalancer that works.

[juan131]

Hi @jmastr

Now set service.type to type LoadBalancer, if your setup allows, NodePort otherwise (this I did not test though) and try using that LB or NodePort directly instead of the dedicated broker's node ports.

For me that fails when using kafkacat. If externalAccess.service.type is LoadBalancer that works.

That's expected because the advertised listeners are configured on a different port (a different one per broker) in the brokers when using externalAccess.service.type=NodePort. Therefore, it's not possible to access using the "default" service.

Please note this is not a bug but a design decision. When setting externalAccess.enabled=true, users are expected to access the cluster using the list of ips/nodeports available to access the brokers (check the screenshot I shared).

[jmastr]

Hey @juan131

That's expected because the advertised listeners are configured on a different port (a different one per broker) in the brokers when using externalAccess.service.type=NodePort. Therefore, it's not possible to access using the "default" service.

Please note this is not a bug but a design decision. When setting externalAccess.enabled=true, users are expected to access the cluster using the list of ips/nodeports available to access the brokers (check the screenshot I shared).

Please have a look at the README:

Following the aforementioned steps will also allow to connect the brokers from the outside using the cluster's default service (when service.type is LoadBalancer or NodePort). Use the property service.externalPort to specify the port used for external connections.

https://github.com/bitnami/charts/blob/194a909268377a6820d91f788c0380f427637ce6/bitnami/kafka/README.md?plain=1#L565

And the aforementioned steps include setting externalAccess.service.type=NodePort. So is the README wrong or a bit ambiguous?

[juan131]

You're right, the README.md is wrong (at least for the "NodePort" case) since it's not compatible. Thanks for pointing it out.

Would you like to send a PR fixing that? I'll be glad to review it.

[jmastr]

@juan131 there you go :)