spring-cloud / stream-applications

Functions and Spring Cloud Stream Applications for data driven microservices
https://spring.io/projects/spring-cloud-stream-applications
250 stars 104 forks source link

Refactor CDC Debezium Source #447

Closed tzolov closed 1 year ago

tzolov commented 1 year ago
corneil commented 1 year ago
  • Add support for DB2, Cassandra, Vitess, Spanner
  • Add AVRO format support. Defaults to JSON
  • Remove the Metadata repo configuration.
  • Remove the Spring property sortcuts. Only the native Debezium properties are used.
  • change the native properties prefix from cdc.config to cdc.debezium.
  • Removes the need to fork and modify the Debezium engine code.
  • Add binding name strategy.
  • Update to latest 2.1.4 Final version.

This is a new Debezium source implementation, build on the Debezium Engine API. I've removed the support for metadata registry as it requires forking and modifying Debezium code. Above changes also means that the new implementation is not backward compatible. This is the reason to add new cdc-debezium-sourc2 and cdc-debezium-spupplier2 instead of modifying the original.

The current implementation is a source and does not implement the contract of functional Supplier. Many of the source implementations provides Spring Integration MessageProducer which is then used to create Supplier<Flux<Message<?>>>

tzolov commented 1 year ago

@corneil the code and description are updated. The implementation does implement the Supplier contract and exposes Supplier<Flux<Message<?>>>.

tzolov commented 1 year ago

Addressed all open review recommendations, including a StepVerifier test for the Supplier<Flux<Message<?>>> debeziumSupplier and updated mongodb testing, though later still require manual testing (description provided in the README).

artembilan commented 1 year ago

Merged as https://github.com/spring-cloud/stream-applications/commit/6cf66ec779faaed28a759dde409b034c92cb7356 after rebasing.

@tzolov , huge thanks: as always tremendous work!

Looking forward for more! 😄