search

awslabs / dynamodb-streams-kinesis-adapter

The Amazon DynamoDB Streams Adapter implements the Amazon Kinesis interface so that your application can use KCL to consume and process data from a DynamoDB stream.
Apache License 2.0
99 stars 37 forks source link

question about shard split #25

Closed vinllen closed 5 years ago

vinllen commented 5 years ago

I still have a question about the shard split. If one partition split into two, will the shard also be split? E.g., there are 4 shards in the partition1, then, partition1 split into partition2 and partition3, my question is will the shards also be split so that there are 4 shards on each new partition? or the old 4 shards on the partition won't be split, only the new stream data will be split into partition1/shard0 and partition2/shard0?

aggarwal commented 5 years ago

Whenever a partition splits into two new partitions, the associated open shard will be closed, and new shards will be created for the new partitions. Any already-closed shards associated with the parent partition will not be split any further.

A partition can only have a single open shard at any given time. Since you did not specify the open/close state of partition1's shards, I will assume that you meant partition1 has 3 closed shards (p1-s0, p1-s1, p1-s2) and a single open shard (p1-s3). When partition1 splits into partition2 and partition3, p1-s3 will be closed, and new shards p2-s0 and p3-s0 will be created in open state. The closed shards p1-s0, p1-s1, p1-s2 will not be modified.

vinllen commented 5 years ago

@aggarwal got, thanks a lot!