Anyone tested with debezium connect?

[bbigras]

It seems to hang while creating a connector.

To reproduce:

git clone https://github.com/bbigras/debezium-examples.git -b redpanda

cd debezium-examples/tutorial

export DEBEZIUM_VERSION=1.4
docker-compose -f docker-compose-sqlserver.yaml up -d

# Start SQL Server connector
cat debezium-sqlserver-init/inventory.sql | docker exec -i tutorial_sqlserver_1 bash -c '/opt/mssql-tools/bin/sqlcmd -U sa -P Password!'

# Start SQL Server connector
curl -i -X POST -H "Accept:application/json" -H  "Content-Type:application/json" http://localhost:8083/connectors/ -d @register-sqlserver.json

Original instructions: https://github.com/debezium/debezium-examples/tree/master/tutorial#using-sql-server

My changes: https://github.com/debezium/debezium-examples/compare/master...bbigras:redpanda?expand=1

[dotnwat]

@bbigras thanks for the report, debezium looks really interesting! do you happen to know if debezium is relying on idempotent producers or kafka transactions directly or indirectly? those are the two core features that will not be in redpanda until march (later this month for a tech preview cc: @rystsov). Otherwise, we would expect it to work and can take a closer look.

[bbigras]

debezium looks really interesting!

Yeah, it seems a very efficient way to get real-time updates from a database.

do you happen to know if debezium is relying on idempotent producers or kafka transactions directly or indirectly?

I have no idea to be honest. I just discovered redpanda/kafka this week because of a comment on HN.

[emaxerrno]

i think @rk3rn3r might know. - do you know if debezium uses txns or idempotent producers?

[rk3rn3r]

Hey @senior7515! Debezium runs on Kafka Connect, which doesn't use tx-producer. If idempotent producer is used depends on the provided producer config which is defined in the Kafka Connect config, but not enabled by default, so that might not be the cause for the KCD (Kafka Connect Distributed) gets stuck on connector creation.

But I confirm Kafka Connect connects nicely to Redpanda, but when creating a connector it get stucked. Recently I got to that same point and didn't have time to continue on it, but maybe next week.

The last log line I see from KCD (Kafka Connect Distributed) was https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L814 "Submitting connector config write request ".

[rk3rn3r]

At KCD startup I saw some ListOffset warnings, although the topic was created by KCD:

2021-02-05 11:33:45,873 WARN   ||  [Consumer clientId=consumer-1-1, groupId=1] Received unknown topic or partition error in ListOffset request for partition my_connect_offsets-5   [org.apache.kafka.clients.consumer.internals.Fetcher]
2021-02-05 11:33:45,873 WARN   ||  [Consumer clientId=consumer-1-1, groupId=1] Received unknown topic or partition error in ListOffset request for partition my_connect_offsets-7   [org.apache.kafka.clients.consumer.internals.Fetcher]
2021-02-05 11:33:45,873 WARN   ||  [Consumer clientId=consumer-1-1, groupId=1] Received unknown topic or partition error in ListOffset request for partition my_connect_offsets-9   [org.apache.kafka.clients.consumer.internals.Fetcher]

and later some other things I found many errors like that:

2021-02-05 12:11:42,244 TRACE  ||  [AdminClient clientId=adminclient-2] Unable to assign Call(callName=listNodes, deadlineMs=1612527162238, tries=0, nextAllowedTryMs=0) to 
a node.   [org.apache.kafka.clients.admin.KafkaAdminClient]

and

2021-02-05 12:11:42,246 DEBUG  ||  [AdminClient clientId=adminclient-2] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 1   [org.apache.kafka.common.network.Selector]
2021-02-05 12:11:42,246 DEBUG  ||  [AdminClient clientId=adminclient-2] Completed connection to node 1. Fetching API versions.   [org.apache.kafka.clients.NetworkClient]
2021-02-05 12:11:42,246 DEBUG  ||  [AdminClient clientId=adminclient-2] Initiating API versions fetch from node 1.   [org.apache.kafka.clients.NetworkClient]
2021-02-05 12:11:42,246 TRACE  ||  [AdminClient clientId=adminclient-2] No version information found when sending API_VERSIONS with correlation id 3 to node 1. Assuming version 3.   [org.apache.kafka.clients.NetworkClient]
2021-02-05 12:11:42,246 DEBUG  ||  [AdminClient clientId=adminclient-2] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=3) and timeout 3600000 to node 1: {client_software_name=apache-kafka-java,client_software_version=2.6.1,_tagged_fields={}}   [org.apache.kafka.clients.NetworkClient]
2021-02-05 12:11:42,246 TRACE  ||  [AdminClient clientId=adminclient-2] KafkaClient#poll retrieved 0 response(s)   [org.apache.kafka.clients.admin.KafkaAdminClient]
2021-02-05 12:11:42,246 TRACE  ||  [AdminClient clientId=adminclient-2] Trying to choose nodes for [] at 1612527102246   [org.apache.kafka.clients.admin.KafkaAdminClient]
2021-02-05 12:11:42,246 TRACE  ||  [AdminClient clientId=adminclient-2] Client is not ready to send to kafka:9092 (id: 1 rack: null). Must delay 9223372036854775807 ms   [org.apache.kafka.clients.admin.KafkaAdminClient]

...

2021-02-05 12:11:42,247 DEBUG  ||  [AdminClient clientId=adminclient-2] Received API_VERSIONS response from node 1 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=2, clientId=adminclient-2, correlationId=4): org.apache.kafka.common.requests.ApiVersionsResponse@2a5c3506   [org.apache.kafka.clients.NetworkClient]

I also don't see a successful JoinGroup (I think it comes after the listNodes call), where it joins the worker group.

[rk3rn3r]

Here's an example docker-compose.yml (but be careful with the high error level):

version: '2'
services:
  kafka:
    image: vectorized/redpanda:latest
    hostname: kafka
    ports:
     - 9092:9092
    command: ["start", "--advertise-kafka-addr", "kafka:9092"]
  postgres:
    image: debezium/example-postgres:1.4
    ports:
     - 5432:5432
    environment:
     - POSTGRES_USER=postgres
     - POSTGRES_PASSWORD=postgres
  connect:
    image: debezium/connect:1.4
    ports:
     - 8083:8083
    links:
     - kafka
     - postgres
    environment:
     - BOOTSTRAP_SERVERS=kafka:9092
     - GROUP_ID=1
     - CONFIG_STORAGE_TOPIC=my_connect_configs
     - OFFSET_STORAGE_TOPIC=my_connect_offsets
     - STATUS_STORAGE_TOPIC=my_connect_statuses
     - LOG_LEVEL=ALL
     - CONNECT_LOG4J_LOGGERS=ALL, stdout, appender
#     - CONNECT_LOG4J_LOGGER_ORG_APACHE_KAFKA_CLIENTS=INFO
     - CONNECT_LOG4J_LOGGER_ORG_ECLIPSE_JETTY=INFO

[dotnwat]

thank you @rk3rn3r for the quick turn around here! this is excellent. it does look like there are some thing we need to investigate w.r.t. compatibility with debezium.

[brennantaylor]

Just to help cut down on some debugging I've attached some debug logs from kafka connect with redpanda and with zk+kafka. 3 brokers in each case to meet the default needs of the 3x replication factor for the topics.

Kafka connect creates 3 topics to store its state. A config, offset and status topic.

The only topic I've seen KC create with redpanda was the offset topic. However it can't seem to create the config or status storage topics. The storage topic has unique constraints (similar to schema registry) where it needs 1 partition to ensure ordered messages since this topic is effectively the database for the configs. It likely uses transactions to commit config changes to this topic as well.

Attached are logs from a kafka-connect pod I'm running locally with a 3 pod redpanda STS. I've turned it up to debug logging with the env var CONNECT_LOG4J_ROOT_LOGLEVEL=DEBUG. Attached is the first 2500 lines of the debug output. In the middle (after boot and plugin loading) you'll see some kafka admin api calls then it seems to fall into a loop reading the offset topic. The loop appears normal.

with redpanda

with kafka

[emaxerrno]

@brennantaylor - thanks for these logs! we'll have idempotent producers and full transactions in a couple of weeks FYI. So should work seamlessly then. watch this ticket #445 (cc @rystsov )

[brennantaylor]

Cool. I'll keep an eye out. Retesting should be a breeze once a docker image is available.

[dotnwat]

@brennantaylor both idempotent producers and transactions are in the tree (under feature flags for a bit longer until they stabilize). @rystsov are there any instructions for turning this on for anyone wishing to test beta?

[emaxerrno]

yeah, is really just there with --set redpanda.enable_idempotence=true

here is an example i just tried on my mac anad it seemed to work as a docker-compose up

  redpanda:
    # TODO: keep updating this version to latest
    image: vectorized/redpanda:v21.6.6
    command: >-
      redpanda start
      --overprovisioned --smp=1 --memory=1G --reserve-memory=0M
      --node-id=0 --check=false
      --set "redpanda.enable_transactions=true"
      --set "redpanda.enable_idempotence=true"
      --advertise-kafka-addr redpanda:9092

@brennantaylor or @bbigras let us know how it goes.

[brennantaylor]

@dotnwat @senior7515 It worked for me.

Was able to spin up:

• a 3 proc red panda cluster
• kafka schema reg (1 internal topic, _schemas 1p3r)
• kafka connect (3 internal topics, config 1p3r, offset 25p3r, status 5p3r)

The kafka connect and schema reg internal topic creation worked with all the defaults.

Kafka connect was able to read the file I made and dump it into the topic I configured both with an without registrying a schema in the schema reg.

[emaxerrno]

@dotnwat @senior7515 It worked for me.

Was able to spin up:

• a 3 proc red panda cluster
• kafka schema reg (1 internal topic, _schemas 1p3r)
• kafka connect (3 internal topics, config 1p3r, offset 25p3r, status 5p3r)

The kafka connect and schema reg internal topic creation worked with all the defaults.

Kafka connect was able to read the file I made and dump it into the topic I configured both with an without registrying a schema in the schema reg.

This is great to hear. 🎉 - a couple of follow up thoughts:

1. schema registry is coming with Avro just around the corner - https://github.com/vectorizedio/redpanda/tree/dev/src/v/pandaproxy/schema_registry

2. we'd love to host you on our /blog so others can learn from you if you are able to share a use case (alex @vectorized .io)

thanks! and glad is up and running!