search

crossplane-contrib / provider-kafka

Crossplane provider for Kafka
Apache License 2.0
24 stars 28 forks source link

When a connection between the Kafka Crossplane provider and MSK drops in AWS, the provider never recovers #62

Closed bogdandisc closed 8 months ago

bogdandisc commented 8 months ago

What happened?

We have the Crossplane Kafka provider deployed on EKS. After a short period it looses the connection to MSK and never recovers. Until this happens I can see the topic being created and synched and I can query it in MSK. After the connection drops the topic CRD gets desynchronised and never recovers as well. Only a full uninstall of the provider, will open the connection again

How can we reproduce it?

  1. Install Crossplane and Kafka provider on EKS
  2. Drop the connection between the provider and MSK
  3. Wait a bit, the connection should drop and never recover.

Setting used for the MSK Kafka secret:

{
     "brokers":[
       "broker-dns:9098"
      ],
      "sasl":{
        "mechanism":"AWS-MSK-IAM"
      }
}

What environment did it happen in?

Crossplane version: 1.13.2 Crossplane Kafka version: 0.4.3 EKS version: 1.24

Kafka provider logs:

[ERROR] unable to initialize sasl; broker: seed 0, err: read tcp 10.2.38.31:35902->10.2.44.218:9098: i/o timeout
[ERROR] unable to initialize sasl; broker: seed 0, err: read tcp 10.2.38.31:51264->10.2.44.218:9098: i/o timeout
[ERROR] unable to initialize sasl; broker: seed 0, err: read tcp 10.2.38.31:41998->10.2.44.218:9098: i/o timeout
[WARN] unable to open connection to broker; addr: broker-dns:9098, broker: seed 0, err: context deadline exceeded
[WARN] unable to open connection to broker; addr: broker-dns:9098, broker: seed 0, err: context deadline exceeded