Is there any option to drop cdc-field on the schema of sink target table ?

databricks / iceberg-kafka-connect

Apache License 2.0

220 stars 49 forks source link

Is there any option to drop cdc-field on the schema of sink target table ? #229

Closed okayhooni closed 7 months ago

okayhooni commented 7 months ago

I think it looks more pretty cdc-field(with value of I & U) is dropped on the target table schema.

But, as you know, it must not drop on SMT chain for connector to processing CDC topic message as expected.

Is there any option to drop cdc-field on the schema of sink target table ?

fqtab commented 7 months ago

This is not possible out of the box today. We might consider adding such a feature in the future but this would be relatively low priority compared to other features in the backlog.

In the meantime, you can try this (I'm not 100% sure it will work):

Either precreate the table without the cdc-field before starting the connector
Or delete the cdc-field from the table after starting the connector

okayhooni commented 7 months ago

Thank you for kind response!

I will close this issue, even though I couldn't apply your suggestion, because I want to sink with iceberg.tables.evolve-schema-enabled option..!