dergyitheron
commented
6 months ago Quick update, I have the following workaround with all pods accessible on dedicated Ingress host names.
The downside of this is that the built-in LoadBalancer is present and unused since I've only used the associated configuration to have correct hostname and port for advertised listener for each node.
I see only few possible solutions from most to least prefered:
- someone here figures it out and tells me something I've missed
- me or someone else suggest PR which solves this (such as being able to use {{ include "common.names.fullname" $ }} in domain or something similar)
- I'll clone the chart and keep my own modified version
- I'll keep the workaround solution
The workaround:
externalAccess:
enabled: true
controller:
service:
type: LoadBalancer
allocateLoadBalancerNodePorts: false
ports:
external: 443
loadBalancerNames:
- "kafka-0.my-domain.org"
- "kafka-1.my-domain.org"
- "kafka-2.my-domain.org"
broker:
service:
type: LoadBalancer
allocateLoadBalancerNodePorts: false
ports:
external: 443
loadBalancerNames:
- "kafka-0.my-domain.org"
- "kafka-1.my-domain.org"
- "kafka-2.my-domain.org"
kraft:
enabled: true
clusterId: 1
zookeeper:
enabled: false
extraDeploy:
- |
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
name: kafka-tls-ingress
namespace: kafka
spec:
entryPoints:
- websecure
routes:
- match: HostSNI(`kafka.my-domain.org`)
services:
- name: kafka
port: 9095
- match: HostSNI(`kafka-0.my-domain.org`)
services:
- name: kafka-controller-0-external-custom
port: 9095
- match: HostSNI(`kafka-1.my-domain.org`)
services:
- name: kafka-controller-1-external-custom
port: 9095
- match: HostSNI(`kafka-2.my-domain.org`)
services:
- name: kafka-controller-2-external-custom
port: 9095
tls:
secretName: kafka-tls-secret
- |
apiVersion: v1
kind: Service
metadata:
name: kafka-controller-0-external-custom
spec:
type: ClusterIP
ports:
- name: tcp-kafka-custom
port: 9095
protocol: TCP
targetPort: external
selector:
app.kubernetes.io/component: controller-eligible
app.kubernetes.io/instance: kafka
app.kubernetes.io/name: kafka
app.kubernetes.io/part-of: kafka
statefulset.kubernetes.io/pod-name: kafka-controller-0
- |
apiVersion: v1
kind: Service
metadata:
name: kafka-controller-1-external-custom
spec:
type: ClusterIP
ports:
- name: tcp-kafka-custom
port: 9095
protocol: TCP
targetPort: external
selector:
app.kubernetes.io/component: controller-eligible
app.kubernetes.io/instance: kafka
app.kubernetes.io/name: kafka
app.kubernetes.io/part-of: kafka
statefulset.kubernetes.io/pod-name: kafka-controller-1
- |
apiVersion: v1
kind: Service
metadata:
name: kafka-controller-2-external-custom
spec:
type: ClusterIP
ports:
- name: tcp-kafka-custom
port: 9095
protocol: TCP
targetPort: external
selector:
app.kubernetes.io/component: controller-eligible
app.kubernetes.io/instance: kafka
app.kubernetes.io/name: kafka
app.kubernetes.io/part-of: kafka
statefulset.kubernetes.io/pod-name: kafka-controller-2I can confirm this works by connecting .NET consumers to Kafka and consuming messages.
carrodher
github-actions[bot]
Name and Version
bitnami/kafka 28.0.3
What architecture are you using?
amd64
What steps will reproduce the bug?
I have a K3s cluster with 3 worker nodes, which have built-in LB (klipper) and Ingress controller (traefik). I have successfully deployed bitnami/kafka helm chart with some custom values listed below. I can connect from within the K3s cluster and produce and consume messages.
Now I'm trying to set up external access including IngressRouteTCP and have some issues with advertised listeners. The consumer connects to the bootstrap server but gets advertised listeners with Kafka ports, not the ingress entrypoint ports.
The traefik entrypoint is on port 443 so I need for the consumer to receive the advertised listener with the external URL set in IngressRouteTCP and port 443.
I've also tried ClusterIP type for external access but that allows only one domain for the service to be specified, so advertised listeners values are even more incorrect (same for every kafka node).
I've also noticed that external access service of type ClusterIP and LoadBalancer creates separate services for each pod, so since I have Ingress on top with the domain for each pod it doesn't really matter which one I use as it ends up in the correct pod every time.
What is the best way to approach this and set up correctly?
Are you using any custom parameters or values?
Values
IngressRouteTCP
What is the expected behavior?
It is unclear to me how I can modify the advertised listeners to return the external listener with the correct domain and correct port for the kafka node.
I expect to see this (for first kafka node, notice the port for EXTERNAL):
I only expect to connect to kafka from inside of the cluster or outside via ingress route but I don't want to expose any additional ports outside.
What do you see instead?
The external listener advertises on port 9095. I cannot set up LoadBalancer type for port 443 because it's already taken by the traefik ingress controller on each K3s node.
extraPortsvalue doesn't help as it tries to allocate the 443 port.Additional information
I can also see this behavior with consumers written in the .NET package Confluent.Kafka.
Connects to the bootstrap server and tries to establish connection on advertised listener found in properties of the kafka node.
Log:
I know I might have some redundant values for the helmchart, I'll take care of that once this is solved. TLS termination is handled by traefik ingress controller and I'm not concerned about plaintext communication within the K3s cluster (nodes are in an isolated network).