javsalgar
commented
4 months ago Hi,
Could you check the rendered configuration that the chart generates? It should be in configmaps and secrets.
Closed acharbha closed 3 months ago
javsalgar
commented
4 months ago Hi,
Could you check the rendered configuration that the chart generates? It should be in configmaps and secrets.
acharbha
commented
4 months ago Following values is working to have stable pods with no restart
helm.exe -n my-kafka get values my-kafka
USER-SUPPLIED VALUES:global:
storageClass: "nfs-lab"
extraConfig: |
allow.everyone.if.no.acl.found=true
super.users=User:admin;User:controller_user;User:dbaassuperadmin
auto.create.topics.enable=TRUE
default.replication.factor=1
delete.topic.enable=FALSE
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
sasl:
client:
users:
- user1
passwords: "user1pass"
controller:
podSecurityContext:
enabled: false
containerSecurityContext:
enabled: false
logPersistence:
enabled: true
automountServiceAccountToken: true
broker:
podSecurityContext:
enabled: false
containerSecurityContext:
enabled: false
logPersistence:
enabled: true
automountServiceAccountToken: true
externalAccess:
enabled: true
autoDiscovery:
enabled: true
controller:
service:
loadBalancerIPs:
- 10.11.XX.Y1
- 10.11.XX.Y2
- 10.11.XX.Y3
publishNotReadyAddresses: true
broker:
service:
publishNotReadyAddresses: true
rbac:
create: true
However, when I start, I try to publish using kafka client its failing SASL auth
kubectl exec --tty -i my-kafka-client --namespace my-kafka -- bash
I have no name!@my-kafka-client:/$ cat /tmp/client.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="user1" \
password="user1pass";
I have no name!@my-kafka-client:/$I have no name!@my-kafka-client:/$ kafka-console-producer.sh \
--producer.config /tmp/client.properties \
--broker-list my-kafka-controller-0.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092,my-kafka-controller-1.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092,my-kafka-controller-2.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092 \
--topic test
>[2024-05-24 09:22:21,948] ERROR [Producer clientId=console-producer] Connection to node -1 (my-kafka-controller-0.my-kafka-controller-headless.my-kafka.svc.cluster.local/10.42.97.8:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-24 09:22:21,949] WARN [Producer clientId=console-producer] Bootstrap broker my-kafka-controller-0.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-24 09:22:22,276] ERROR [Producer clientId=console-producer] Connection to node -2 (my-kafka-controller-1.my-kafka-controller-headless.my-kafka.svc.cluster.local/10.42.203.111:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-24 09:22:22,276] WARN [Producer clientId=console-producer] Bootstrap broker my-kafka-controller-1.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092 (id: -2 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-24 09:22:22,591] ERROR [Producer clientId=console-producer] Connection to node -3 (my-kafka-controller-2.my-kafka-controller-headless.my-kafka.svc.cluster.local/10.42.205.225:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)Even with plaintext protocol I cannot produce the messages getting error -
listeners:
client:
protocol: PLAINTEXT
controller:
protocol: PLAINTEXT
interbroker:
protocol: PLAINTEXT
external:
protocol: PLAINTEXTkafka-console-producer.sh \
--broker-list my-kafka-controller-0.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092,my-kafka-controller-1.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092,my-kafka-controller-2.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092 \
--topic test
>[2024-05-24 09:36:58,131] WARN [Producer clientId=console-producer] Connection to node -3 (my-kafka-controller-2.my-kafka-controller-headless.my-kafka.svc.cluster.local/10.42.205.33:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-05-24 09:36:58,131] WARN [Producer clientId=console-producer] Bootstrap broker my-kafka-controller-2.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092 (id: -3 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-24 09:36:58,139] WARN [Producer clientId=console-producer] Connection to node -1 (my-kafka-controller-0.my-kafka-controller-headless.my-kafka.svc.cluster.local/10.42.97.164:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-05-24 09:36:58,139] WARN [Producer clientId=console-producer] Bootstrap broker my-kafka-controller-0.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-24 09:36:58,247] WARN [Producer clientId=console-producer] Connection to node -2 (my-kafka-controller-1.my-kafka-controller-headless.my-kafka.svc.cluster.local/10.42.203.186:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-05-24 09:36:58,248] WARN [Producer clientId=console-producer] Bootstrap broker my-kafka-controller-1.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092 (id: -2 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)There's something basic which is not working - javsalgar. I built kafka with all default value still can't produce the messages -
PS C:\Users\acharbha> helm -n my-kafka install my-kafka bitnami/kafka
NAME: my-kafka
LAST DEPLOYED: Fri May 24 19:22:30 2024
NAMESPACE: my-kafka
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: kafka
CHART VERSION: 28.2.4
APP VERSION: 3.7.0
** Please be patient while the chart is being deployed **
Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:
my-kafka.my-kafka.svc.cluster.local
Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:
my-kafka-controller-0.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092
my-kafka-controller-1.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092
my-kafka-controller-2.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092
The CLIENT listener for Kafka client connections from within your cluster have been configured with the following security settings:
- SASL authentication
To connect a client to your Kafka, you need to create the 'client.properties' configuration files with the content below:
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="user1" \
password="$(kubectl get secret my-kafka-user-passwords --namespace my-kafka -o jsonpath='{.data.client-passwords}' | base64 -d | cut -d , -f 1)";
To create a pod that you can use as a Kafka client run the following commands:
kubectl run my-kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.7.0-debian-12-r6 --namespace my-kafka --command -- sleep infinity
kubectl cp --namespace my-kafka /path/to/client.properties my-kafka-client:/tmp/client.properties
kubectl exec --tty -i my-kafka-client --namespace my-kafka -- bash
PRODUCER:
kafka-console-producer.sh \
--producer.config /tmp/client.properties \
--broker-list my-kafka-controller-0.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092,my-kafka-controller-1.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092,my-kafka-controller-2.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092 \
--topic test
CONSUMER:
kafka-console-consumer.sh \
--consumer.config /tmp/client.properties \
--bootstrap-server my-kafka.my-kafka.svc.cluster.local:9092 \
--topic test \
--from-beginning
WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs:
- controller.resources
+info https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/.\kubectl.exe get pods -n my-kafka
NAME READY STATUS RESTARTS AGE
my-kafka-client 1/1 Running 0 11h
my-kafka-controller-0 1/1 Running 0 16m
my-kafka-controller-1 1/1 Running 0 16m
my-kafka-controller-2 1/1 Running 0 16mkubectl get secret my-kafka-user-passwords --namespace my-kafka -o jsonpath='{.data.client-passwords}' | base64 -d | cut -d , -f 1
CvqPI061lNkubectl exec --tty -i my-kafka-client --namespace my-kafka -- bash
I have no name!@my-kafka-client:/$ cat /tmp/client.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="user1" \
password="CvqPI061lN";
root@acharbha-mobl1:~# kubectl exec --tty -i my-kafka-client --namespace my-kafka -- bash
I have no name!@my-kafka-client:/$ kafka-console-producer.sh \
--producer.config /tmp/client.properties \
--broker-list my-kafka-controller-0.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092,my-kafka-controller-1.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092,my-kafka-controller-2.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092 \
--topic test
>[2024-05-24 14:07:47,063] ERROR [Producer clientId=console-producer] Connection to node -3 (my-kafka-controller-2.my-kafka-controller-headless.my-kafka.svc.cluster.local/10.42.205.55:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-24 14:07:47,063] WARN [Producer clientId=console-producer] Bootstrap broker my-kafka-controller-2.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092 (id: -3 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-24 14:07:47,072] ERROR [Producer clientId=console-producer] Connection to node -2 (my-kafka-controller-1.my-kafka-controller-headless.my-kafka.svc.cluster.local/10.42.203.157:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-24 14:07:47,072] WARN [Producer clientId=console-producer] Bootstrap broker my-kafka-controller-1.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092 (id: -2 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-24 14:07:47,387] ERROR [Producer clientId=console-producer] Connection to node -1 (my-kafka-controller-0.my-kafka-controller-headless.my-kafka.svc.cluster.local/10.42.97.186:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-24 14:07:47,387] WARN [Producer clientId=console-producer] Bootstrap broker my-kafka-controller-0.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-24 14:07:47,697] ERROR [Producer clientId=console-producer] Connection to node -3 (my-kafka-controller-2.my-kafka-controller-headless.my-kafka.svc.cluster.local/10.42.205.55:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
kubectl exec --tty -i my-kafka-controller-0 --namespace my-kafka -- bash
/opt/bitnami/kafka/bin$ kafka-metadata-quorum.sh --bootstrap-server my-kafka-controller-0.my-kafka-controller-headless.my-kafka.svc.cluster.local:9092 describe --status
org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at org.apache.kafka.tools.MetadataQuorumCommand.handleDescribeStatus(MetadataQuorumCommand.java:210)
at org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:111)
at org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:62)
at org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:57)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
fmulero
commented
4 months ago Hi @acharbha
I am trying to reproduce your issue. At the moment I've seen that the documentation about the value sasl.users.passwords seems not valid. That value should be set in this manner (with a list):
sasl:
client:
users:
- user1
passwords:
- "user1pass"I hope to come back soon with more news.
acharbha
commented
4 months ago I appreciate your offer to help with this. I expect that at least the default configuration should be in a working state. https://github.com/bitnami/charts/issues/26387#issuecomment-2129629390
acharbha
commented
4 months ago @fmulero any update ?
github-actions[bot]
commented
4 months ago This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.
fmulero
commented
4 months ago Sorry about my very late response.
About the problem you reported here with the default values, Did you removed previous PV? I am not able to reproduce that problem and I think you have a conflict coming from existing PV.
I also tried to reproduce the original issue with latest version of the chart and I am not able to do it, not sure if changes applied on #27097 could fix this issue also. Could you give it a try?
github-actions[bot]
commented
3 months ago Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.
Name and Version
bitnami/kafka 28.3.0
What architecture are you using?
amd64
What steps will reproduce the bug?
kubectl create namespace my-kafkahelm -n my-kafka install my-kafka oci://registry-1.docker.io/bitnamicharts/kafka -f .\my-kafka\values.ymlAre you using any custom parameters or values?
Yes
What is the expected behavior?
Zero restarts of pods.
What do you see instead?
controller pods restarts always
Additional information
.\kubectl.exe logs -n my-kafka my-kafka-controller-0 -c kafka -f