search

awslabs / spark-sql-kinesis-connector

Spark Structured Streaming Kinesis Data Streams connector supports both GetRecords and SubscribeToShard (Enhanced Fan-Out, EFO)
Apache License 2.0
26 stars 13 forks source link

Consideration for default value for `kinesis.maxFetchTimePerShardSec` #17

Open MJFND opened 6 months ago

MJFND commented 6 months ago

I spent lot of time in debugging and optimization especially migrating from a Databricks connector.

I think its worth to have a default value for kinesis.maxFetchTimePerShardSec.

hwanghw commented 6 months ago

Thanks for your comments. I agree a default value for kinesis.maxFetchTimePerShardSec can prevent the executor from running for long run before reach kinesis.maxFetchRecordsPerShard especially when the event arriving rate is low.

As the change of setting default value for kinesis.maxFetchTimePerShardSec is not backward compatible, I tend to make the change together with next major release.

MJFND commented 6 months ago

Thanks, sounds good.

On Thu, May 23, 2024, 7:21 PM hwanghw @.***> wrote:

Thanks for your comments. I agree a default value for kinesis.maxFetchTimePerShardSec can prevent the executor from running for long run before reach kinesis.maxFetchRecordsPerShard especially when the event arriving rate is low.

As the change of setting default value for kinesis.maxFetchTimePerShardSec is not backward compatible, I tend to make the change together with next major release.

— Reply to this email directly, view it on GitHub https://github.com/awslabs/spark-sql-kinesis-connector/issues/17#issuecomment-2128184050, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC65XLUGTETHL2UNPMULNFTZDZ2W3AVCNFSM6AAAAABHY55TEGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRYGE4DIMBVGA . You are receiving this because you authored the thread.Message ID: @.***>