Support Cassandra CDC logical types (new option decodeCDCDataTypes)

datastax / pulsar-sink

An Apache Pulsar® sink for transferring events/messages from Pulsar topics to Apache Cassandra®, DataStax Astra or DataStax Enterprise (DSE) tables.

Apache License 2.0

14 stars 10 forks source link

Closed aymkhalil closed 2 years ago

aymkhalil commented 2 years ago

This patch will handle avro logical types coming from an upstream CSC. The following C* types will be handled:

The implementation relies on the ConvertingCodecs from C* (StringTo[BigDecimal|BigInteger|CqlDuration] ConvertingCodecs.

I will follow up with a patch with a flag to disable this conversion from non-CSC users.

aymkhalil commented 2 years ago

In the latest patch:

I changed the logical type check to respect the schema and avoid reflection
Update the schemas in the test to use optional logical type schema (i.e. Union with null) to match the upstream CDC
Added a decodeCDCDataTypes flag so the feature is turned off altogether for existing users to reduce regression chances