search

awslabs / amazon-kinesis-client

Client library for Amazon Kinesis
Apache License 2.0
644 stars 467 forks source link

Can KCL support Heterogeneous workers ? #103

Open darkcrawler01 opened 8 years ago

darkcrawler01 commented 8 years ago

I am investigating if its possible to diversify the ec2 InstanceType of the instances running the KCL workers to take advantage of the EC2 Spot Market, that would require simultaneously running KCL on instances with different cores/memory. As far as I can see KCL uniformly distributes the shards across all active workers. There is currently no way to specify a weight to be associated with each worker. Workers could specify a weight W between 1-10 which would mean that worker with W10 should be assigned 10x more shards than Worker with W1. Is this the proper way to handling this problem ? Or is there any other alternative solution like using dockers to homogenize the heterogeneous compute pools ?

pfifer commented 8 years ago

You're correct that the KCL really doesn't support heterogeneous fleets.

We agree with the change you have suggested, and will prioritize it accordingly against the other customer requests we receive.

Others - Please comment or add your reaction to this post so our team is aware of how to prioritize this change.

zerth commented 7 years ago

This would be helpful, especially if the weighting were automatic. E.G., a worker unable to keep up on 2 shards wouldn't steal leases from a worker staying caught up on 10. (For me, this is an issue with deploying workers as services in an ECS cluster running multiple different services with uneven container distribution; some workers are consistently able to burst above their CPU reservations on certain hosts).

kalyanaramansanthanam commented 7 years ago

any progress or workaround this feature?

kalyanaramansanthanam commented 6 years ago

any progress on this feature request?

abhishek-khare commented 5 years ago

We are using KCL 1.9.1 with autoscaling groups, and would really like to use a mix of instance types. Right now we are restricted to use instance types with same core and memory configuration. It would be great if this can be supported with 1.x versions as well.