search

redpanda-data / console

Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging.
https://redpanda.com
3.76k stars 346 forks source link

Got "connect: connection refused" for Kafka Connect created by Strimzi #692

Closed hongbo-miao closed 1 year ago

hongbo-miao commented 1 year ago

First, just want to thank you for this awesome tool! Even can view Avro format messages! 😃

I installed Redpanda Console by

helm upgrade \
  redpanda-console \
  console \
  --install \
  --repo=https://charts.redpanda.com \
  --namespace=hm-redpanda-console \
  --create-namespace \
  --values=my-values.yaml

my-values.yaml:

console:
  config:
    kafka:
      brokers:
        - hm-kafka-kafka-bootstrap.hm-kafka.svc:9092
      schemaRegistry:
        enabled: true
        urls:
          - http://confluent-schema-registry.hm-confluent-schema-registry.svc:8081
    connect:
      enabled: true
      clusters:
        - name: hm-kafka-iot-kafka-connect
          url: http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083

However, I got error connecting my Kafka Connect which is created by Strimzi in the Redpanda Console pod log:

{"level":"info","ts":"2023-05-06T05:12:24.156Z","msg":"started Redpanda Console","version":"v2.2.3","built_at":"1679491937"}
{"level":"info","ts":"2023-05-06T05:12:24.156Z","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata"}
{"level":"info","ts":"2023-05-06T05:12:24.165Z","msg":"successfully connected to kafka cluster","advertised_broker_count":1,"topic_count":12,"controller_id":0,"kafka_version":"v3.4"}
{"level":"info","ts":"2023-05-06T05:12:24.165Z","msg":"creating schema registry client and testing connectivity"}
{"level":"info","ts":"2023-05-06T05:12:24.175Z","msg":"successfully tested schema registry connectivity"}
{"level":"info","ts":"2023-05-06T05:12:24.176Z","msg":"creating Kafka connect HTTP clients and testing connectivity to all clusters"}
{"level":"warn","ts":"2023-05-06T05:12:24.182Z","msg":"connect cluster is not reachable","cluster_name":"hm-kafka-iot-kafka-connect","cluster_address":"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083","error":"Get \"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083/\": dial tcp 10.43.193.144:8083: connect: connection refused"}
{"level":"info","ts":"2023-05-06T05:12:24.182Z","msg":"tested Kafka connect cluster connectivity","successful_clusters":0,"failed_clusters":1}
{"level":"info","ts":"2023-05-06T05:12:24.182Z","msg":"successfully create Kafka connect service"}
{"level":"info","ts":"2023-05-06T05:12:24.552Z","msg":"Server listening on address","address":"[::]:8080","port":8080}
{"level":"warn","ts":"2023-05-06T05:13:05.781Z","msg":"failed to list connectors from Kafka connect cluster","cluster_name":"hm-kafka-iot-kafka-connect","cluster_address":"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083","error":"Get \"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083/connectors?expand=info&expand=status\": dial tcp 10.43.193.144:8083: connect: connection refused"}
{"level":"error","ts":"2023-05-06T05:13:05.807Z","msg":"Sending REST error","cluster_name":"hm-kafka-iot-kafka-connect","route":"/api/kafka-connect/clusters/hm-kafka-iot-kafka-connect","method":"GET","status_code":503,"remote_address":"127.0.0.1:43902","public_error":"Failed to get cluster info: Get \"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083/\": dial tcp 10.43.193.144:8083: connect: connection refused","error":"Get \"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083/\": dial tcp 10.43.193.144:8083: connect: connection refused"}
{"level":"error","ts":"2023-05-06T05:13:07.660Z","msg":"Sending REST error","cluster_name":"hm-kafka-iot-kafka-connect","route":"/api/kafka-connect/clusters/hm-kafka-iot-kafka-connect","method":"GET","status_code":503,"remote_address":"127.0.0.1:43902","public_error":"Failed to get cluster info: Get \"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083/\": dial tcp 10.43.193.144:8083: connect: connection refused","error":"Get \"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083/\": dial tcp 10.43.193.144:8083: connect: connection refused"}
{"level":"error","ts":"2023-05-06T05:13:08.350Z","msg":"Sending REST error","cluster_name":"hm-kafka-iot-kafka-connect","route":"/api/kafka-connect/clusters/hm-kafka-iot-kafka-connect","method":"GET","status_code":503,"remote_address":"127.0.0.1:43902","public_error":"Failed to get cluster info: Get \"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083/\": dial tcp 10.43.193.144:8083: connect: connection refused","error":"Get \"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083/\": dial tcp 10.43.193.144:8083: connect: connection refused"}
{"level":"info","ts":"2023-05-06T05:36:56.166Z","msg":"immediate metadata update triggered","source":"kafka_client","why":"querying metadata for consumer initialization"}
{"level":"info","ts":"2023-05-06T05:36:56.173Z","msg":"assigning partitions","source":"kafka_client","why":"new assignments from direct consumer","how":"assigning everything new, keeping current assignment","input":"hm.motor[0{1099950 e-1 ce0}]"}
{"level":"error","ts":"2023-05-06T05:36:56.192Z","msg":"errors while fetching records","topic_name":"","partition":0,"error":"context canceled"}
{"level":"warn","ts":"2023-05-06T05:38:03.425Z","msg":"failed to list connectors from Kafka connect cluster","cluster_name":"hm-kafka-iot-kafka-connect","cluster_address":"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083","error":"Get \"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083/connectors?expand=info&expand=status\": dial tcp 10.43.193.144:8083: connect: connection refused"}
{"level":"error","ts":"2023-05-06T05:38:03.460Z","msg":"Sending REST error","cluster_name":"hm-kafka-iot-kafka-connect","route":"/api/kafka-connect/clusters/hm-kafka-iot-kafka-connect","method":"GET","status_code":503,"remote_address":"127.0.0.1:34500","public_error":"Failed to get cluster info: Get \"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083/\": dial tcp 10.43.193.144:8083: connect: connection refused","error":"Get \"http://hm-kafka-iot-kafka-connect-connect-api.hm-kafka:8083/\": dial tcp 10.43.193.144:8083: connect: connection refused"}

Also, in the UI, it shows

image

However, if I port-forward Kafka Connect locally, I can reach to it by

# Open first terminal
kubectl port-forward service/hm-kafka-iot-kafka-connect-connect-api --namespace=hm-kafka 8083:8083
Forwarding from 127.0.0.1:8083 -> 8083
Forwarding from [::1]:8083 -> 8083
Handling connection for 8083

# Open second terminal
➜ curl --location 'http://localhost:8083/connectors?expand=info&expand=status'
{"hm-motor-jdbc-sink-kafka-connector":{"info":{"name":"hm-motor-jdbc-sink-kafka-connector","config":{"connector.class":"io.confluent.connect.jdbc.JdbcSinkConnector","table.name.format":"motor","transforms.convertTimestamp.target.type":"Timestamp","connection.password":"${file:/opt/kafka/external-configuration/hm-iot-db-credentials-volume/iot-db-credentials.properties:timescaledb_password}","topics":"hm.motor","tasks.max":"8","batch.size":"100000","transforms":"convertTimestamp","transforms.convertTimestamp.type":"org.apache.kafka.connect.transforms.TimestampConverter$Value","value.converter.schema.registry.url":"http://confluent-schema-registry.hm-confluent-schema-registry.svc:8081","transforms.convertTimestamp.field":"timestamp","connection.user":"${file:/opt/kafka/external-configuration/hm-iot-db-credentials-volume/iot-db-credentials.properties:timescaledb_user}","name":"hm-motor-jdbc-sink-kafka-connector","connection.url":"jdbc:postgresql://timescale.hm-timescale.svc:5432/hm_iot_db","value.converter":"io.confluent.connect.avro.AvroConverter","insert.mode":"insert","pk.mode":"record_value","pk.fields":"timestamp"},"tasks":[{"connector":"hm-motor-jdbc-sink-kafka-connector","task":0},{"connector":"hm-motor-jdbc-sink-kafka-connector","task":1},{"connector":"hm-motor-jdbc-sink-kafka-connector","task":2},{"connector":"hm-motor-jdbc-sink-kafka-connector","task":3},{"connector":"hm-motor-jdbc-sink-kafka-connector","task":4},{"connector":"hm-motor-jdbc-sink-kafka-connector","task":5},{"connector":"hm-motor-jdbc-sink-kafka-connector","task":6},{"connector":"hm-motor-jdbc-sink-kafka-connector","task":7}],"type":"sink"},"status":{"name":"hm-motor-jdbc-sink-kafka-connector","connector":{"state":"RUNNING","worker_id":"10.42.0.21:8083"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"10.42.0.21:8083"},{"id":1,"state":"RUNNING","worker_id":"10.42.0.18:8083"},{"id":2,"state":"RUNNING","worker_id":"10.42.0.20:8083"},{"id":3,"state":"RUNNING","worker_id":"10.42.0.19:8083"},{"id":4,"state":"RUNNING","worker_id":"10.42.0.22:8083"},{"id":5,"state":"RUNNING","worker_id":"10.42.0.21:8083"},{"id":6,"state":"RUNNING","worker_id":"10.42.0.18:8083"},{"id":7,"state":"RUNNING","worker_id":"10.42.0.20:8083"}],"type":"sink"}}}

If need, here is my Kafka Connect YAML

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: hm-kafka-iot-kafka-connect
  namespace: hm-kafka
  annotations:
    # use-connector-resources configures this KafkaConnect
    # to use KafkaConnector resources to avoid
    # needing to call the Connect REST API directly
    strimzi.io/use-connector-resources: "true"
spec:
  image: ghcr.io/hongbo-miao/hm-kafka-iot-kafka-connect:latest
  replicas: 5
  bootstrapServers: hm-kafka-kafka-bootstrap:9093
  tls:
    trustedCertificates:
      - secretName: hm-kafka-cluster-ca-cert
        certificate: ca.crt
  config:
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1
    config.providers: file
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
  externalConfiguration:
    volumes:
      - name: hm-iot-db-credentials-volume
        secret:
          secretName: hm-iot-db-credentials

and my Kafka Connect Dockerfile

FROM docker.io/alpine:3.17.3 AS builder
USER root:root
RUN mkdir -p /opt/kafka/plugins/ \
  # https://www.confluent.io/hub/confluentinc/kafka-connect-avro-converter
   && wget --no-verbose --output-document=kafka-connect-avro-converter.zip https://d1i4a15mxbxib1.cloudfront.net/api/plugins/confluentinc/kafka-connect-avro-converter/versions/7.3.3/confluentinc-kafka-connect-avro-converter-7.3.3.zip \
   && mkdir -p /opt/kafka/plugins/kafka-connect-avro-converter/ \
   && unzip kafka-connect-avro-converter.zip -d /opt/kafka/plugins/kafka-connect-avro-converter/ \
   && rm -f kafka-connect-avro-converter.zip \
  # confluentinc-kafka-connect-jdbc
  # https://www.confluent.io/hub/confluentinc/kafka-connect-elasticsearch
  && wget --no-verbose --output-document=confluentinc-kafka-connect-jdbc.zip https://d1i4a15mxbxib1.cloudfront.net/api/plugins/confluentinc/kafka-connect-jdbc/versions/10.7.1/confluentinc-kafka-connect-jdbc-10.7.1.zip \
  && mkdir -p /opt/kafka/plugins/confluentinc-kafka-connect-jdbc/ \
  && unzip confluentinc-kafka-connect-jdbc.zip -d /opt/kafka/plugins/confluentinc-kafka-connect-jdbc/ \
  && rm -f confluentinc-kafka-connect-jdbc.zip
USER 1001

FROM quay.io/strimzi/kafka:0.34.0-kafka-3.4.0
USER root:root
COPY --from=builder /opt/kafka/plugins/ /opt/kafka/plugins/
USER 1001

Any idea? Thanks!

weeco commented 1 year ago

Try running the curl with the exact URL that is printed from inside the Console container by execing into it. This error is coming from Go's stdlib which has always been right so far.

Maybe the DNS address is wrong, you haven't posted your service manifest here and I don't know what service strimzi would create.

hongbo-miao commented 1 year ago

Thanks @weeco ! curl is good way to test!

I posted the solution at https://stackoverflow.com/a/76191353/2000548 It was because Kafka Connect created by Strimzi has network policy limitation, and I need create a new one for Redpanda Console.

BTW, the Redpanda Console UI quality is soooooo good! ☺️

image
weeco commented 1 year ago

Thanks for reporting back your solution, also glad you like Console!