search

awslabs / spark-sql-kinesis-connector

Spark Structured Streaming Kinesis Data Streams connector supports both GetRecords and SubscribeToShard (Enhanced Fan-Out, EFO)
Apache License 2.0
26 stars 13 forks source link

Plumb through Record TTL to KPL #31

Closed penniman26 closed 1 month ago

penniman26 commented 1 month ago

I'm looking to add an optional arg, record TTL, to plumb through to the KPL. The purpose of this is that I don't want my stream writer to drop any records (by default, it will retry only up to 30 seconds). Would you be open to a PR to plumb this through?

hwanghw commented 1 month ago

Changes merged to main branch. Thanks for your contribution.