datastax / pulsar-sink

An Apache Pulsar® sink for transferring events/messages from Pulsar topics to Apache Cassandra®, DataStax Astra or DataStax Enterprise (DSE) tables.
Apache License 2.0
14 stars 10 forks source link

Allow mapping headers/properties to non string columns #23

Open jclu123 opened 2 years ago

jclu123 commented 2 years ago

I would like to map one of the property from CSC's data topic: properties:[writetime=1647625263548354] to a real timestamp type (or a long) so it can be properly saved as as a timestamp in the sink cassandra database. Something like ...my_ts = toTimeStamp(headers.writetime)... would be good.

While I have considered using my_ts=now() as a substitute, there is, however, a little discrepancy, time wise, between write-time written by CSC and now() taken by the Pulsar-sink, and would like to get around that as much as I can.