Aiven-Open / http-connector-for-apache-kafka

Apache Kafka Connect sink connector for HTTP
Apache License 2.0
88 stars 44 forks source link

add support to handle records with null value #183

Closed bionarkos closed 11 months ago

bionarkos commented 11 months ago

tombstone records currently cause the sink-task to stop, as an exception is thrown. Added a new config property "behavior.on.null.values" to control this behavior in a more granular fashion. The following enum values are allowed:

jeqo commented 11 months ago

@bionarkos thanks for the PR! Though I would like to validate first if this feature is actually needed.

Have you considered using transforms (a.k.a SMT) to filter1 tombstones2?

For instance, by adding:

{
    //...
    "transforms": "filter",
    "transforms.filter.type": "org.apache.kafka.connect.transforms.Filter",
    "transforms.filter.predicate": "isTombstone",
    "predicates": "isTombstone",
    "predicates.isTombstone.type": "org.apache.kafka.connect.transforms.predicates.RecordIsTo
mbstone"
}

you may get the same behavior without changes on the connector

bionarkos commented 11 months ago

Thank you very much @jeqo for the good tip, the filtering possibility solved my issue, so I can close this PR. :bow: