search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
9.03k stars 9.22k forks source link

[bitnami/kafka] kafka and kafka-connect pods end in CrashLoopBackoff and no zookeeper pod #17895

Closed ghost closed 1 year ago

ghost commented 1 year ago

Name and Version

kafka-23.0.5 3.5.0

What architecture are you using?

amd64

What steps will reproduce the bug?

Docker version

Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:35:18 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:18 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

KinD version

kind v0.20.0 go1.20.4 linux/amd64

Helm version

version.BuildInfo{Version:"v3.12.1", GitCommit:"f32a527a060157990e2aa86bf45010dfb3cc8b8d", GitTreeState:"clean", GoVersion:"go1.20.4"}

Deploy mongodb service and client. Create a database and user.

$ helm install mongodb bitnami/mongodb --set auth.rootPassword=mongodbrootpassword

$ kubectl run --namespace default mongodb-client --rm --tty -i --restart='Never' --image docker.io/bitnami/mongodb:4.2.8-debian-10-r39 --command -- bash
mongo admin --host "mongodb" --authenticationDatabase admin -u root -p mongodbrootpassword
> use mydb
switched to db mydb
> db.createUser(
... {
...   user: "user",
...   pwd: "mongodbuserpassword",
...   roles: [ { role: "dbOwner", db: "mydb" } ]
... }
... )

Build kafka-connect image and push to DockerHub (see below for Dockerfile used)

docker build -t dkjackyu/mykafka:1.0 -f Dockerfile . 
docker push dkjackyu/mykafka:1.0

Deploy kafka and kafka-connect (see below for values.yml)

helm install kafka bitnami/kafka -f values.yml

Reference:

Are you using any custom parameters or values?

kafka-connect Dockerfile

FROM bitnami/kafka:latest

RUN mkdir -p /opt/bitnami/kafka/plugins && \
    cd /opt/bitnami/kafka/plugins && \
    curl --remote-name --location --silent https://search.maven.org/remotecontent?filepath=org/mongodb/kafka/mongo-kafka-connect/1.2.0/mongo-kafka-connect-1.2.0-all.jar

CMD /opt/bitnami/kafka/bin/connect-standalone.sh /opt/bitnami/kafka/config/connect-standalone.properties /opt/bitnami/kafka/config/mongo.properties

values.yml

auth:
  clientProtocol: sasl
  interBrokerProtocol: plaintext
  jaas:
    clientUser: kafkausername
    clientPassword: kafkapassword
extraDeploy:
  - |
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: {{ include "common.names.fullname" . }}-connect
      labels: {{- include "common.labels.standard" . | nindent 4 }}
        app.kubernetes.io/component: connector
    spec:
      replicas: 1
      selector:
        matchLabels: {{- include "common.labels.matchLabels" . | nindent 6 }}
          app.kubernetes.io/component: connector
      template:
        metadata:
          labels: {{- include "common.labels.standard" . | nindent 8 }}
            app.kubernetes.io/component: connector
        spec:
          containers:
            - name: connect
              image: dkjackyu/mykafka:1.0
              imagePullPolicy: IfNotPresent
              command:
                - /bin/bash
                - -ec
                - |
                  /opt/bitnami/kafka/bin/connect-standalone.sh /config/connect-standalone.properties /config/mongodb.properties
              ports:
                - name: connector
                  containerPort: 8083
              volumeMounts:
                - name: configuration
                  mountPath: /config
          volumes:
            - name: configuration
              configMap:
                name: {{ include "common.names.fullname" . }}-connect
  - |
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: {{ include "common.names.fullname" . }}-connect
      labels: {{- include "common.labels.standard" . | nindent 4 }}
        app.kubernetes.io/component: connector
    data:
      connect-standalone.properties: |-
        bootstrap.servers = {{ include "common.names.fullname" . }}-0.{{ include "common.names.fullname" . }}-headless.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}:{{ .Values.service.port }}
        key.converter=org.apache.kafka.connect.json.JsonConverter
        value.converter=org.apache.kafka.connect.json.JsonConverter
        key.converter.schemas.enable=true
        value.converter.schemas.enable=true
        offset.storage.file.filename=/tmp/connect.offsets
        offset.flush.interval.ms=10000
        plugin.path=/opt/bitnami/kafka/plugins
        sasl.mechanism=PLAIN
        security.protocol=SASL_PLAINTEXT
        sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
          username="kafkausername" \
          password="kafkapassword";
        consumer.sasl.mechanism=PLAIN
        consumer.security.protocol=SASL_PLAINTEXT
        consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
          username="kafkausername" \
          password="kafkapassword";
      mongodb.properties: |-
        connection.uri=mongodb://user:mongodbrootpassword@mongodb.default.svc.cluster.local:27017/mydb
        name=mongo-sink
        topics=mytopic
        connector.class=com.mongodb.kafka.connect.MongoSinkConnector
        tasks.max=1
        key.converter=org.apache.kafka.connect.json.JsonConverter
        value.converter=org.apache.kafka.connect.json.JsonConverter
        key.converter.schemas.enable=false
        value.converter.schemas.enable=false
        database=mydb
        collection=sink
  - |
    apiVersion: v1
    kind: Service
    metadata:
      name: {{ include "common.names.fullname" . }}-connect
      labels: {{- include "common.labels.standard" . | nindent 4 }}
        app.kubernetes.io/component: connector
    spec:
      ports:
        - protocol: TCP
          port: 8083
          targetPort: connector
      selector: {{- include "common.labels.matchLabels" . | nindent 4 }}
        app.kubernetes.io/component: connector

What is the expected behavior?

Expected to see a mongodb pod, a kafka pod, a kafka-connect pod and a zookeeper pod running.

What do you see instead?

There's no zookeeper, and the kafka and kafka-connect pods fail:

NAME                                 READY   STATUS             RESTARTS        AGE
pod/kafka-0                          0/1     CrashLoopBackOff   11 (2m5s ago)   34m
pod/kafka-connect-797fd8b758-nrxfb   0/1     CrashLoopBackOff   11 (99s ago)    34m
pod/mongodb-98f78dd-s87c6            1/1     Running            0               5h44m

NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
service/kafka            ClusterIP   10.96.179.68   <none>        9092/TCP                     34m
service/kafka-connect    ClusterIP   10.96.99.184   <none>        8083/TCP                     34m
service/kafka-headless   ClusterIP   None           <none>        9092/TCP,9094/TCP,9093/TCP   34m
service/mongodb          ClusterIP   10.96.141.53   <none>        27017/TCP                    5h44m

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kafka-connect   0/1     1            0           34m
deployment.apps/mongodb         1/1     1            1           5h44m

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/kafka-connect-797fd8b758   1         1         0       34m
replicaset.apps/mongodb-98f78dd            1         1         1       5h44m

NAME                     READY   AGE
statefulset.apps/kafka   0/1     34m

$ kubectl logs pod/kafka-0
kafka 15:18:53.27 
kafka 15:18:53.27 Welcome to the Bitnami kafka container
kafka 15:18:53.27 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 15:18:53.28 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 15:18:53.28 
kafka 15:18:53.28 INFO  ==> ** Starting Kafka setup **
kafka 15:18:53.37 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
kafka 15:18:53.38 INFO  ==> Initializing Kafka...
kafka 15:18:53.39 INFO  ==> No injected configuration files found, creating default config files
kafka 15:18:53.99 INFO  ==> Configuring Kafka for inter-broker communications with PLAINTEXT authentication.
kafka 15:18:54.00 WARN  ==> Inter-broker communications are configured as PLAINTEXT. This is not safe for production environments.
kafka 15:18:54.00 INFO  ==> Configuring Kafka for client communications with SASL_PLAINTEXT authentication.
kafka 15:18:54.02 INFO  ==> Generating JAAS authentication file
kafka 15:18:54.03 INFO  ==> Creating users in Zookeeper
$ kubectl describe pod/kafka-0
Name:             kafka-0
Namespace:        tanzu
Priority:         0
Service Account:  kafka
Node:             poland-control-plane/172.18.0.2
Start Time:       Tue, 25 Jul 2023 15:46:44 +0100
Labels:           app.kubernetes.io/component=kafka
                  app.kubernetes.io/instance=kafka
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=kafka
                  controller-revision-hash=kafka-5df5888d88
                  helm.sh/chart=kafka-23.0.5
                  statefulset.kubernetes.io/pod-name=kafka-0
Annotations:      checksum/jaas-secret: f61d9fadee98a75e921795db952e292d1b88d1e439ea9cf045d4f38cc695da00
Status:           Running
IP:               10.244.0.18
IPs:
  IP:           10.244.0.18
Controlled By:  StatefulSet/kafka
Containers:
  kafka:
    Container ID:  containerd://9c5fddb7998d7a51d73f0b0a2d28dbb3b7520110982897e903fcca3514a6edfb
    Image:         docker.io/bitnami/kafka:3.5.0-debian-11-r21
    Image ID:      docker.io/bitnami/kafka@sha256:fb7c104952ccd1c2adff62a643e15109213c322b1f8bcdd9a4ed685233331745
    Ports:         9092/TCP, 9094/TCP, 9093/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Command:
      /scripts/setup.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 25 Jul 2023 16:23:59 +0100
      Finished:     Tue, 25 Jul 2023 16:24:02 +0100
    Ready:          False
    Restart Count:  12
    Liveness:       tcp-socket :kafka-client delay=10s timeout=5s period=10s #success=1 #failure=3
    Readiness:      tcp-socket :kafka-client delay=5s timeout=5s period=10s #success=1 #failure=6
    Environment:
      BITNAMI_DEBUG:                                       false
      MY_POD_IP:                                            (v1:status.podIP)
      MY_POD_NAME:                                         kafka-0 (v1:metadata.name)
      KAFKA_CFG_ZOOKEEPER_CONNECT:                         
      KAFKA_INTER_BROKER_LISTENER_NAME:                    INTERNAL
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP:            INTERNAL:PLAINTEXT,CLIENT:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_CFG_SASL_ENABLED_MECHANISMS:                   PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
      KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL:      PLAIN
      KAFKA_CFG_LISTENERS:                                 INTERNAL://:9094,CLIENT://:9092,CONTROLLER://:9093
      KAFKA_CFG_ADVERTISED_LISTENERS:                      INTERNAL://$(MY_POD_NAME).kafka-headless.tanzu.svc.cluster.local:9094,CLIENT://$(MY_POD_NAME).kafka-headless.tanzu.svc.cluster.local:9092
      ALLOW_PLAINTEXT_LISTENER:                            yes
      KAFKA_OPTS:                                          -Djava.security.auth.login.config=/opt/bitnami/kafka/config/kafka_jaas.conf
      KAFKA_CLIENT_USERS:                                  user
      KAFKA_CLIENT_PASSWORDS:                              <set to the key 'client-passwords' in secret 'kafka-jaas'>  Optional: false
      KAFKA_ZOOKEEPER_PROTOCOL:                            PLAINTEXT
      KAFKA_VOLUME_DIR:                                    /bitnami/kafka
      KAFKA_LOG_DIR:                                       /opt/bitnami/kafka/logs
      KAFKA_CFG_DELETE_TOPIC_ENABLE:                       false
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE:                 true
      KAFKA_HEAP_OPTS:                                     -Xmx1024m -Xms1024m
      KAFKA_CFG_LOG_FLUSH_INTERVAL_MESSAGES:               10000
      KAFKA_CFG_LOG_FLUSH_INTERVAL_MS:                     1000
      KAFKA_CFG_LOG_RETENTION_BYTES:                       1073741824
      KAFKA_CFG_LOG_RETENTION_CHECK_INTERVAL_MS:           300000
      KAFKA_CFG_LOG_RETENTION_HOURS:                       168
      KAFKA_CFG_MESSAGE_MAX_BYTES:                         1000012
      KAFKA_CFG_LOG_SEGMENT_BYTES:                         1073741824
      KAFKA_CFG_LOG_DIRS:                                  /bitnami/kafka/data
      KAFKA_CFG_DEFAULT_REPLICATION_FACTOR:                1
      KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR:          1
      KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR:  1
      KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR:             1
      KAFKA_CFG_NUM_IO_THREADS:                            8
      KAFKA_CFG_NUM_NETWORK_THREADS:                       3
      KAFKA_CFG_NUM_PARTITIONS:                            1
      KAFKA_CFG_NUM_RECOVERY_THREADS_PER_DATA_DIR:         1
      KAFKA_CFG_SOCKET_RECEIVE_BUFFER_BYTES:               102400
      KAFKA_CFG_SOCKET_REQUEST_MAX_BYTES:                  104857600
      KAFKA_CFG_SOCKET_SEND_BUFFER_BYTES:                  102400
      KAFKA_CFG_ZOOKEEPER_CONNECTION_TIMEOUT_MS:           6000
      KAFKA_CFG_AUTHORIZER_CLASS_NAME:                     
      KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND:            true
      KAFKA_CFG_SUPER_USERS:                               User:admin
      KAFKA_ENABLE_KRAFT:                                  true
      KAFKA_KRAFT_CLUSTER_ID:                              kafka_cluster_id_test1
      KAFKA_CFG_PROCESS_ROLES:                             broker,controller
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES:                 CONTROLLER
    Mounts:
      /bitnami/kafka from data (rw)
      /opt/bitnami/kafka/logs from logs (rw)
      /scripts/setup.sh from scripts (rw,path="setup.sh")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rzgm6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-kafka-0
    ReadOnly:   false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kafka-scripts
    Optional:  false
  logs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-rzgm6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  38m                   default-scheduler  Successfully assigned tanzu/kafka-0 to poland-control-plane
  Normal   Pulled     36m (x5 over 38m)     kubelet            Container image "docker.io/bitnami/kafka:3.5.0-debian-11-r21" already present on machine
  Normal   Created    36m (x5 over 38m)     kubelet            Created container kafka
  Normal   Started    36m (x5 over 38m)     kubelet            Started container kafka
  Warning  BackOff    3m8s (x164 over 37m)  kubelet            Back-off restarting failed container kafka in pod kafka-0_tanzu(06c1da05-ba35-45f9-9b22-3fe840f7cb72)
$ kubectl logs pod/kafka-connect-797fd8b758-nrxfb
[2023-07-25 15:24:33,137] INFO Kafka Connect worker initializing ... (org.apache.kafka.connect.cli.AbstractConnectCli:114)
[2023-07-25 15:24:33,146] INFO WorkerInfo values: 
    jvm.args = -Xms256M, -Xmx2G, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -XX:MaxInlineLevel=15, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/opt/bitnami/kafka/bin/../logs, -Dlog4j.configuration=file:/opt/bitnami/kafka/bin/../config/connect-log4j.properties
    jvm.spec = BellSoft, OpenJDK 64-Bit Server VM, 17.0.7, 17.0.7+7-LTS
    jvm.classpath = /opt/bitnami/kafka/bin/../libs/activation-1.1.1.jar:/opt/bitnami/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/bitnami/kafka/bin/../libs/audience-annotations-0.13.0.jar:/opt/bitnami/kafka/bin/../libs/commons-cli-1.4.jar:/opt/bitnami/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/bitnami/kafka/bin/../libs/connect-api-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-basic-auth-extension-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-json-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-mirror-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-mirror-client-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-runtime-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-transforms-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/hk2-api-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/hk2-locator-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/hk2-utils-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/jackson-annotations-2.13.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-core-2.13.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-databind-2.13.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-dataformat-csv-2.13.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-datatype-jdk8-2.13.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs-base-2.13.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs-json-provider-2.13.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-jaxb-annotations-2.13.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-scala_2.12-2.13.5.jar:/opt/bitnami/kafka/bin/../libs/jakarta.activation-api-1.2.2.jar:/opt/bitnami/kafka/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/bitnami/kafka/bin/../libs/jakarta.inject-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/bitnami/kafka/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/bitnami/kafka/bin/../libs/jakarta.xml.bind-api-2.3.3.jar:/opt/bitnami/kafka/bin/../libs/javassist-3.29.2-GA.jar:/opt/bitnami/kafka/bin/../libs/javax.activation-api-1.2.0.jar:/opt/bitnami/kafka/bin/../libs/javax.annotation-api-1.3.2.jar:/opt/bitnami/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/bitnami/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/bitnami/kafka/bin/../libs/jaxb-api-2.3.1.jar:/opt/bitnami/kafka/bin/../libs/jersey-client-2.39.1.jar:/opt/bitnami/kafka/bin/../libs/jersey-common-2.39.1.jar:/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-2.39.1.jar:/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-core-2.39.1.jar:/opt/bitnami/kafka/bin/../libs/jersey-hk2-2.39.1.jar:/opt/bitnami/kafka/bin/../libs/jersey-server-2.39.1.jar:/opt/bitnami/kafka/bin/../libs/jetty-client-9.4.51.v20230217.jar:/opt/bitnami/kafka/bin/../libs/jetty-continuation-9.4.51.v20230217.jar:/opt/bitnami/kafka/bin/../libs/jetty-http-9.4.51.v20230217.jar:/opt/bitnami/kafka/bin/../libs/jetty-io-9.4.51.v20230217.jar:/opt/bitnami/kafka/bin/../libs/jetty-security-9.4.51.v20230217.jar:/opt/bitnami/kafka/bin/../libs/jetty-server-9.4.51.v20230217.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlet-9.4.51.v20230217.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlets-9.4.51.v20230217.jar:/opt/bitnami/kafka/bin/../libs/jetty-util-9.4.51.v20230217.jar:/opt/bitnami/kafka/bin/../libs/jetty-util-ajax-9.4.51.v20230217.jar:/opt/bitnami/kafka/bin/../libs/jline-3.22.0.jar:/opt/bitnami/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/bitnami/kafka/bin/../libs/jose4j-0.9.3.jar:/opt/bitnami/kafka/bin/../libs/kafka-clients-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-group-coordinator-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-log4j-appender-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-metadata-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-raft-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-server-common-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-shell-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-storage-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-storage-api-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-examples-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-scala_2.12-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-test-utils-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-tools-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-tools-api-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka_2.12-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/lz4-java-1.8.0.jar:/opt/bitnami/kafka/bin/../libs/maven-artifact-3.8.4.jar:/opt/bitnami/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/bitnami/kafka/bin/../libs/metrics-core-4.1.12.1.jar:/opt/bitnami/kafka/bin/../libs/netty-buffer-4.1.92.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-codec-4.1.92.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-common-4.1.92.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-handler-4.1.92.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-resolver-4.1.92.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-4.1.92.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-classes-epoll-4.1.92.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-native-epoll-4.1.92.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-native-unix-common-4.1.92.Final.jar:/opt/bitnami/kafka/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/bitnami/kafka/bin/../libs/paranamer-2.8.jar:/opt/bitnami/kafka/bin/../libs/plexus-utils-3.3.0.jar:/opt/bitnami/kafka/bin/../libs/reflections-0.9.12.jar:/opt/bitnami/kafka/bin/../libs/reload4j-1.2.25.jar:/opt/bitnami/kafka/bin/../libs/rocksdbjni-7.1.2.jar:/opt/bitnami/kafka/bin/../libs/scala-collection-compat_2.12-2.10.0.jar:/opt/bitnami/kafka/bin/../libs/scala-java8-compat_2.12-1.0.2.jar:/opt/bitnami/kafka/bin/../libs/scala-library-2.12.15.jar:/opt/bitnami/kafka/bin/../libs/scala-logging_2.12-3.9.4.jar:/opt/bitnami/kafka/bin/../libs/scala-reflect-2.12.15.jar:/opt/bitnami/kafka/bin/../libs/slf4j-api-1.7.36.jar:/opt/bitnami/kafka/bin/../libs/slf4j-reload4j-1.7.36.jar:/opt/bitnami/kafka/bin/../libs/snappy-java-1.1.10.0.jar:/opt/bitnami/kafka/bin/../libs/swagger-annotations-2.2.8.jar:/opt/bitnami/kafka/bin/../libs/trogdor-3.5.0.jar:/opt/bitnami/kafka/bin/../libs/zookeeper-3.6.4.jar:/opt/bitnami/kafka/bin/../libs/zookeeper-jute-3.6.4.jar:/opt/bitnami/kafka/bin/../libs/zstd-jni-1.5.5-1.jar
    os.spec = Linux, amd64, 5.19.0-46-generic
    os.vcpus = 4
 (org.apache.kafka.connect.runtime.WorkerInfo:71)
[2023-07-25 15:24:33,148] INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.AbstractConnectCli:120)
[2023-07-25 15:24:33,169] INFO Loading plugin from: /opt/bitnami/kafka/plugins/mongo-kafka-connect-1.10.1-all.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:275)
[2023-07-25 15:24:33,983] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/bitnami/kafka/plugins/mongo-kafka-connect-1.10.1-all.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:297)
[2023-07-25 15:24:33,983] INFO Added plugin 'com.mongodb.kafka.connect.MongoSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:33,984] INFO Added plugin 'com.mongodb.kafka.connect.MongoSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:33,984] INFO Added plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:33,984] INFO Added plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:33,985] INFO Added plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,404] INFO Registered loader: jdk.internal.loader.ClassLoaders$AppClassLoader@7ad041f3 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:297)
[2023-07-25 15:24:36,404] INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,405] INFO Added plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,405] INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,406] INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,406] INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,407] INFO Added plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,407] INFO Added plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,408] INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,408] INFO Added plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,408] INFO Added plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,409] INFO Added plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,409] INFO Added plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,410] INFO Added plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,410] INFO Added plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,411] INFO Added plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,411] INFO Added plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,412] INFO Added plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,412] INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,412] INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,413] INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,413] INFO Added plugin 'org.apache.kafka.connect.transforms.Filter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,413] INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,414] INFO Added plugin 'org.apache.kafka.connect.transforms.HeaderFrom$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,414] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,414] INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,415] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,415] INFO Added plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,415] INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,416] INFO Added plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,416] INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,416] INFO Added plugin 'org.apache.kafka.connect.transforms.DropHeaders' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,417] INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,417] INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,417] INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,418] INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,418] INFO Added plugin 'org.apache.kafka.connect.transforms.InsertHeader' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,418] INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,419] INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,419] INFO Added plugin 'org.apache.kafka.connect.transforms.HeaderFrom$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,419] INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,423] INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,424] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,424] INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,425] INFO Added plugin 'org.apache.kafka.connect.transforms.predicates.HasHeaderKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,425] INFO Added plugin 'org.apache.kafka.connect.transforms.predicates.RecordIsTombstone' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,425] INFO Added plugin 'org.apache.kafka.connect.transforms.predicates.TopicNameMatches' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,426] INFO Added plugin 'org.apache.kafka.common.config.provider.FileConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,426] INFO Added plugin 'org.apache.kafka.common.config.provider.DirectoryConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,426] INFO Added plugin 'org.apache.kafka.common.config.provider.EnvVarConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,427] INFO Added plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:230)
[2023-07-25 15:24:36,428] INFO Added aliases 'MongoSinkConnector' and 'MongoSink' to plugin 'com.mongodb.kafka.connect.MongoSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,428] INFO Added aliases 'MongoSourceConnector' and 'MongoSource' to plugin 'com.mongodb.kafka.connect.MongoSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,429] INFO Added aliases 'MirrorCheckpointConnector' and 'MirrorCheckpoint' to plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,429] INFO Added aliases 'MirrorHeartbeatConnector' and 'MirrorHeartbeat' to plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,429] INFO Added aliases 'MirrorSourceConnector' and 'MirrorSource' to plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,430] INFO Added aliases 'MockSinkConnector' and 'MockSink' to plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,430] INFO Added aliases 'MockSourceConnector' and 'MockSource' to plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,430] INFO Added aliases 'SchemaSourceConnector' and 'SchemaSource' to plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,431] INFO Added aliases 'VerifiableSinkConnector' and 'VerifiableSink' to plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,431] INFO Added aliases 'VerifiableSourceConnector' and 'VerifiableSource' to plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,431] INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,444] INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,444] INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,444] INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,445] INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,445] INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,445] INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,445] INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,446] INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,446] INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,446] INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,446] INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,447] INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,447] INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,447] INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,447] INFO Added aliases 'SimpleHeaderConverter' and 'Simple' to plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,447] INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,448] INFO Added alias 'DropHeaders' to plugin 'org.apache.kafka.connect.transforms.DropHeaders' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:505)
[2023-07-25 15:24:36,448] INFO Added alias 'Filter' to plugin 'org.apache.kafka.connect.transforms.Filter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:505)
[2023-07-25 15:24:36,449] INFO Added alias 'InsertHeader' to plugin 'org.apache.kafka.connect.transforms.InsertHeader' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:505)
[2023-07-25 15:24:36,449] INFO Added alias 'RegexRouter' to plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:505)
[2023-07-25 15:24:36,449] INFO Added alias 'TimestampRouter' to plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:505)
[2023-07-25 15:24:36,449] INFO Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:505)
[2023-07-25 15:24:36,450] INFO Added alias 'HasHeaderKey' to plugin 'org.apache.kafka.connect.transforms.predicates.HasHeaderKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:505)
[2023-07-25 15:24:36,450] INFO Added alias 'RecordIsTombstone' to plugin 'org.apache.kafka.connect.transforms.predicates.RecordIsTombstone' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:505)
[2023-07-25 15:24:36,450] INFO Added alias 'TopicNameMatches' to plugin 'org.apache.kafka.connect.transforms.predicates.TopicNameMatches' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:505)
[2023-07-25 15:24:36,450] INFO Added alias 'BasicAuthSecurityRestExtension' to plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:505)
[2023-07-25 15:24:36,450] INFO Added aliases 'AllConnectorClientConfigOverridePolicy' and 'All' to plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,451] INFO Added aliases 'NoneConnectorClientConfigOverridePolicy' and 'None' to plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,451] INFO Added aliases 'PrincipalConnectorClientConfigOverridePolicy' and 'Principal' to plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:508)
[2023-07-25 15:24:36,488] INFO StandaloneConfig values: 
    access.control.allow.methods = 
    access.control.allow.origin = 
    admin.listeners = null
    auto.include.jmx.reporter = true
    bootstrap.servers = [kafka-0.kafka-headless.tanzu.svc.cluster.local:]
    client.dns.lookup = use_all_dns_ips
    config.providers = []
    connector.client.config.override.policy = All
    header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter
    key.converter = class org.apache.kafka.connect.json.JsonConverter
    listeners = [http://:8083]
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    offset.flush.interval.ms = 10000
    offset.flush.timeout.ms = 5000
    offset.storage.file.filename = /tmp/connect.offsets
    plugin.path = [/opt/bitnami/kafka/plugins]
    response.http.headers.config = 
    rest.advertised.host.name = null
    rest.advertised.listener = null
    rest.advertised.port = null
    rest.extension.classes = []
    ssl.cipher.suites = null
    ssl.client.auth = none
    ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
    ssl.endpoint.identification.algorithm = https
    ssl.engine.factory.class = null
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.certificate.chain = null
    ssl.keystore.key = null
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLSv1.3
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.certificates = null
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    task.shutdown.graceful.timeout.ms = 5000
    topic.creation.enable = true
    topic.tracking.allow.reset = true
    topic.tracking.enable = true
    value.converter = class org.apache.kafka.connect.json.JsonConverter
 (org.apache.kafka.connect.runtime.standalone.StandaloneConfig:369)
[2023-07-25 15:24:36,489] INFO Creating Kafka admin client (org.apache.kafka.connect.runtime.WorkerConfig:261)
[2023-07-25 15:24:36,496] INFO AdminClientConfig values: 
    auto.include.jmx.reporter = true
    bootstrap.servers = [kafka-0.kafka-headless.tanzu.svc.cluster.local:]
    client.dns.lookup = use_all_dns_ips
    client.id = 
    connections.max.idle.ms = 300000
    default.api.timeout.ms = 60000
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retries = 2147483647
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = [hidden]
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.connect.timeout.ms = null
    sasl.login.read.timeout.ms = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.login.retry.backoff.max.ms = 10000
    sasl.login.retry.backoff.ms = 100
    sasl.mechanism = PLAIN
    sasl.oauthbearer.clock.skew.seconds = 30
    sasl.oauthbearer.expected.audience = null
    sasl.oauthbearer.expected.issuer = null
    sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
    sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
    sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
    sasl.oauthbearer.jwks.endpoint.url = null
    sasl.oauthbearer.scope.claim.name = scope
    sasl.oauthbearer.sub.claim.name = sub
    sasl.oauthbearer.token.endpoint.url = null
    security.protocol = SASL_PLAINTEXT
    security.providers = null
    send.buffer.bytes = 131072
    socket.connection.setup.timeout.max.ms = 30000
    socket.connection.setup.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
    ssl.endpoint.identification.algorithm = https
    ssl.engine.factory.class = null
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.certificate.chain = null
    ssl.keystore.key = null
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLSv1.3
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.certificates = null
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
 (org.apache.kafka.clients.admin.AdminClientConfig:369)
[2023-07-25 15:24:36,538] ERROR Stopping due to error (org.apache.kafka.connect.cli.AbstractConnectCli:100)
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
    at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:551)
    at org.apache.kafka.clients.admin.Admin.create(Admin.java:144)
    at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:262)
    at org.apache.kafka.connect.runtime.WorkerConfig.kafkaClusterId(WorkerConfig.java:393)
    at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:124)
    at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:94)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:112)
Caused by: org.apache.kafka.common.config.ConfigException: Invalid url in bootstrap.servers: kafka-0.kafka-headless.tanzu.svc.cluster.local:
    at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:59)
    at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48)
    at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:508)
    ... 6 more
$ kubectl describe pod/kafka-connect-797fd8b758-nrxfb
Name:             kafka-connect-797fd8b758-nrxfb
Namespace:        tanzu
Priority:         0
Service Account:  default
Node:             poland-control-plane/172.18.0.2
Start Time:       Tue, 25 Jul 2023 15:46:44 +0100
Labels:           app.kubernetes.io/component=connector
                  app.kubernetes.io/instance=kafka
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=kafka
                  helm.sh/chart=kafka-23.0.5
                  pod-template-hash=797fd8b758
Annotations:      <none>
Status:           Running
IP:               10.244.0.17
IPs:
  IP:           10.244.0.17
Controlled By:  ReplicaSet/kafka-connect-797fd8b758
Containers:
  connect:
    Container ID:  containerd://5d5f720b6ecae26003548a8db8069d4f4db2322146bc0a1024ed7dd2eb2c7884
    Image:         dkjackyu/mykafka:1.0
    Image ID:      docker.io/dkjackyu/mykafka@sha256:f86deff847241a7b739624549fc83c0fc2fdfe70c14768408f01e4c640cb92d6
    Port:          8083/TCP
    Host Port:     0/TCP
    Command:
      /bin/bash
      -ec
      /opt/bitnami/kafka/bin/connect-standalone.sh /config/connect-standalone.properties /config/mongodb.properties

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Tue, 25 Jul 2023 16:24:32 +0100
      Finished:     Tue, 25 Jul 2023 16:24:36 +0100
    Ready:          False
    Restart Count:  12
    Environment:    <none>
    Mounts:
      /config from configuration (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j8s2l (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  configuration:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kafka-connect
    Optional:  false
  kube-api-access-j8s2l:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  41m                   default-scheduler  Successfully assigned tanzu/kafka-connect-797fd8b758-nrxfb to poland-control-plane
  Normal   Pulled     39m (x5 over 41m)     kubelet            Container image "dkjackyu/mykafka:1.0" already present on machine
  Normal   Created    39m (x5 over 41m)     kubelet            Created container connect
  Normal   Started    39m (x5 over 41m)     kubelet            Started container connect
  Warning  BackOff    104s (x178 over 41m)  kubelet            Back-off restarting failed container connect in pod kafka-connect-797fd8b758-nrxfb_tanzu(d68e6982-e7f0-47e6-8f50-65931e3b499b)

Additional information

Following this article: https://tanzu.vmware.com/developer/guides/kafka-mongodb/ and trying to reproduce what's in there.

javsalgar commented 1 year ago

Hi,

There is currently a major refactor of the kafka chart that should most of the issues:

https://github.com/bitnami/charts/pull/17507

MisterTimn commented 1 year ago

Could this issue just be caused by the fact that kraft is enabled by default since 22.0.0? In any case you will need to set kraft.enabled=false and enable zookeeper to proceed, even after the refactoring I believe.

migruiz4 commented 1 year ago

Hi there!

We have released a new major of the bitnami/kafka chart (24.x.x) which refactors the chart architecture and configuration.

Could you please confirm if this issue persists in the latest version?

You can find all the changes introduced in this version as well as some guidance on how to upgrade in the Upgrading notes

ghost commented 1 year ago

Thanks, will give it a try.

github-actions[bot] commented 1 year ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

ghost commented 1 year ago

Could this issue just be caused by the fact that kraft is enabled by default since 22.0.0? In any case you will need to set kraft.enabled=false and enable zookeeper to proceed, even after the refactoring I believe.

@MisterTimn How do I do this? The 'values.yml' (see above) I'm currently using is from an example I'm following, and I don't see any setting related to Kraft or zookeeper.

github-actions[bot] commented 1 year ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 1 year ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.