jcustenborder / kafka-connect-transform-common

Common Transforms for Kafka Connect.
Apache License 2.0
154 stars 59 forks source link

Allow TimestampNowField to output TTL for DynamoDB #101

Open acemoo opened 1 year ago

acemoo commented 1 year ago

This adds the following options to the TimestampNowField transformer:

All of these options have sane defaults to ensure backward compatibility. The idea behind the first two is that DynamoDB expects a TTL value per item inserted. The idea behind the last option is that the current implementation outputs time since epoch in milliseconds. DynamoDB expects a TTL value in Unix time, which is the time since epoch in seconds.

The current ValidEnum and ConfigUtils::getEnum don't agree with each other on which field to use for the ChronoUnit enum. One uses ChronoUnit::name the other uses the enum value. We could handle this on 2 ways.

  1. Move ValidChronoUnit to connect-utils this would mean we use values like "DAYS".
  2. Make ValidEnum and ConfigUtils::getEnum play nice with ChronoUnit this would mean we use values like "Days" From my limited experience with adding my own enum for the target type, it seems option 1 would be more in line with existing things.