search

spring-attic / cdc-debezium

Change Data Capture (CDC) source that captures and streams change events from various databases. Leverages on Debezium and supports MySQL, PostgreSQL, MongoDB, Oracle and SQL Server databases.
Apache License 2.0
52 stars 15 forks source link

Different types of converter support #12

Open 4whomtbts opened 3 years ago

4whomtbts commented 3 years ago

It seems that this project currently doesn't support varying converter type(eg. Avro, Protobuf) and It only supports JSON converter. Is there any way to change converter type?

Thank you for creating and maintaining cool project :)

tzolov commented 3 years ago

Hi @4whomtbts , this is correct. Current CDC source version is (pre)configured with only for JSON type. Honestly I don't remember what stopped me back then to enable support for the other exchange type. Perhaps it has something to do with the fact that we deliberately use the embedded debezium engine and therefore might not rely on the Kafka converter (just guessing).

Either way if you can share your reasons/needs to have such support i might try to explore how to enable multiple exchange formats

tzolov commented 3 years ago

Btw @4whomtbts you might want to switch to the new CDC source version here: https://github.com/spring-cloud/stream-applications/tree/master/applications/source/cdc-debezium-source thought the type support is still the same there.

Can you (re)submit the issue to support the multiple types in the new (https://github.com/spring-cloud/stream-applications) github repository? If we are to enable this it will happen there.

4whomtbts commented 3 years ago

Hi @4whomtbts , this is correct. Current CDC source version is (pre)configured with only for JSON type. Honestly I don't remember what stopped me back then to enable support for the other exchange type. Perhaps it has something to do with the fact that we deliberately use the embedded debezium engine and therefore might not rely on the Kafka converter (just guessing).

Either way if you can share your reasons/needs to have such support i might try to explore how to enable multiple exchange formats

@tzolov
Thank your for your reply :) I needed to vary converter type to use Avro with schema registry. Since I'm currently working on our own stream sink to achieve real-time database replication/sync, as Kafka connect does. So, I'm creating wrapper/adapter of jdbc-sink project, in the project we are making use of Avro and Confluent schema registry to ensure integrity of schema and support compression.

I read this project's code and I though that I could make some patch to enable varying converter type. I made my own patch that makes this project to support configuring converter name and schema registry URL. It works well so far 😃 . As you said, I'll resubmit this issue on the repo that you mentioned.

Oh, actually I have a another question. Does this project support option for sending "schema change" message to binder? It seems that It doesn't. As far as I know, Kafka Connect Debezium doesn't send schema change messages to same topic as row data changes are kept. That is possible, because Kafka Connect takes responsibility of routing messages to topic. But because of this project doesn't run on Kafka Connect, and It sends messages to topic itself, I end up receive schema changes + row data. That's why I want this project could support options for turning off sending schema change to same binder(Because both Debezium, Kafka Connect are big and complicating project, I'm not pretty sure, If I'm talking about arguable/correct issue. Currently, I think I need to remove or filter schema change messages in sink. But I think It would be good, If I could turn off sending schema change messages)

Thank you!