Flux needs a "bucketCountToPoll" config separate from the current "bucketCount" config, which tells Flux how many buckets exist for polling purposes; enforce that bucketCountToPoll is either >= bucketCount, or unset (default = bucketCount). The current bucketCount config will continue to control the number of buckets that will be used when submitting new workflow executions.
This way, users can deploy a code change to safely reduce the number of buckets submitted to, while still polling for any remaining work on the other buckets.
It may also be beneficial to allow bucketCount and bucketCountToPoll to be dynamically adjusted at runtime via new methods on FluxCapacitorImpl (not the FluxCapacitor interface), so users can manage scaling bucket counts from their application code without having to deploy a code change.
When task list bucketing is enabled (see https://github.com/danielgmyers/flux-swf-client/blob/main/flux/src/main/java/software/amazon/aws/clients/swf/flux/TaskListConfig.java#L88 for more information) it is currently not safe to lower the number of buckets if that level of bucketing is no longer necessary, since as long as new workflows may be submitted to a bucket, that bucket needs to be polled for work, and Flux only polls for work on buckets it can submit to.
Flux needs a "bucketCountToPoll" config separate from the current "bucketCount" config, which tells Flux how many buckets exist for polling purposes; enforce that bucketCountToPoll is either >= bucketCount, or unset (default = bucketCount). The current bucketCount config will continue to control the number of buckets that will be used when submitting new workflow executions.
This way, users can deploy a code change to safely reduce the number of buckets submitted to, while still polling for any remaining work on the other buckets.
It may also be beneficial to allow
bucketCountandbucketCountToPollto be dynamically adjusted at runtime via new methods onFluxCapacitorImpl(not theFluxCapacitorinterface), so users can manage scaling bucket counts from their application code without having to deploy a code change.