brendan-p-lynch
commented
8 months ago How are you identifying the number of shards that are being processed on each worker?
A pattern we have seen in the past is that a child shard will have a lease that is being taken by a worker, and that they are dispersed onto the workers with less shards. Because workers don't have a distinction of which shards are active and which are not the active shards are not always taken evenly.
As of right now there is no configuration to improve the distribution of shards.
In the Dynamo table that holds the leases and checkpoint information, I can see that workers have an uneven distribution of shards. In my case for example - for 256 shards and 41 workers (pods in Kubernetes), the majority have ~6-8 shards assigned, but I also see ~5 workers with only 1-2 shards.
How can I make the distribution more evenly? is there some configuration I can change for it?
This causes workers to not fully utilize their resources (the workers with the 1-2 shards barely use any CPU and I would rather have another worker with ~6-8 shards than 5 workers with 1-2 shards each).
It is worth mentioning I am using the amazon-kinesis-client-python library.