[Question] Accessing Kafka from outside of Kubernetes cluster - misconfiguration

nikhileshva commented 4 years ago

So, I've configured the Strimzi Kafka external listeners as type nodeport. The connection test is successful. However, when a message to pushed to the topic, Kafka always returns with this - Topic MyTopic not present in metadata after 60000 ms. , or sometimes this - Expiring 1 record(s) for MyTopic-0:120026 ms has passed since batch creation

Here are the configs:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: clusterName
spec:
  kafka:
    version: 2.4.0
    replicas: 1
    listeners:
      plain: {}
      tls: {}
      external:
        type: nodeport
        tls: false
        overrides:
          bootstrap:
            nodePort: 31194
          brokers:
            - broker: 0
              nodePort: 31195
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "2.4"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        deleteClaim: false
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

There's no authentication or authorization involved for now. However, I did try using scram-sha-512 with a KafkaUser. And whenever a message was sent to Kafka, I could see this in logs:

│ 2020-05-20 11:49:19,290 INFO Processing notification(s) to /config/changes (kafka.common.ZkNodeChangeNotificationListener) [/config/changes-event-process-thread]                                                                        │
│ 2020-05-20 11:49:19,294 INFO Processing override for entityPath: users/myUser with config: Map(SCRAM-SHA-512 -> [hidden]) (kafka.server.DynamicConfigManager) [/config/changes-event-process-thread]                                      │
│ 2020-05-20 11:49:19,294 INFO Removing PRODUCE quota for user myUser (kafka.server.ClientQuotaManager) [/config/changes-event-process-thread]                                                                                              │
│ 2020-05-20 11:49:19,294 INFO Removing FETCH quota for user myUser (kafka.server.ClientQuotaManager) [/config/changes-event-process-thread]                                                                                                │
│ 2020-05-20 11:49:19,294 INFO Removing REQUEST quota for user myUser (kafka.server.ClientRequestQuotaManager) [/config/changes-event-process-thread]                                                                                       │
│ 2020-05-20 11:49:19,302 INFO Processing notification(s) to /config/changes (kafka.common.ZkNodeChangeNotificationListener) [/config/changes-event-process-thread]

And the topic I created in cli (with no user/acls) - assuming anyone can push to those topics.

I'm really not sure where something went wrong while configuring.

scholzj commented 4 years ago

I do not think the messahes anotu the quota etc. are related to sending a message. Have you deleted the KafkaUser resource after you tried it? If not, it will still exist and be periodically reconciled.

As for the topic: How did you created the topic? Can you also share how are you trying to send the message to the topic? How do you connect etc.?

scholzj commented 4 years ago

@nikhileshva Did you managed to resolve this? Can we close the issue?

nikhileshva commented 4 years ago

Yes, we can close it. Thanks.

strimzi / strimzi-kafka-operator

[Question] Accessing Kafka from outside of Kubernetes cluster - misconfiguration #3077