awslabs / amazon-kinesis-client

Client library for Amazon Kinesis
Apache License 2.0
642 stars 465 forks source link

Retrieving shard consumer's current lease's hash range key #1249

Open apoorvt95 opened 8 months ago

apoorvt95 commented 8 months ago

Is it possible to retrieve current lease's hash range key in the initialize lifecycle method of KCL consumer?

The only way I could find is through injecting LeaseCoordinator dependency in the ShardRecordProcessor implementation. This ends up creating a circular dependency during dependency injection.

LeaseCoordinator <- Schedulor ConfigBuilder <- ShardRecordFactory <- ShardRecordProcessor <- LeaseCoordinator.

brendan-p-lynch commented 7 months ago

Could you explain the use case that you are looking for in getting the hash key range for a lease in the initialize function?

apoorvt95 commented 6 months ago

I have a use case where I'm pushing data to Kinesis streams and partitioning my data deterministically.

I'm reading the data using KCL library. In my use case, I have metadata stored in a Datastore partitioned over range of hash keys for Kinesis Shards. I want to selectively load the metadata for a specific shard using the shard's hash range key during initialize function of a Shard consumer.