search

aws / amazon-mwaa-docker-images

Apache License 2.0
27 stars 12 forks source link

Letting each condition control its frequency #150

Closed ashishgo-aws closed 1 month ago

ashishgo-aws commented 1 month ago

Using throttling, each condition was being checked once per minute. But a condition may need to perform its check at a different pace like for example the TaskMonitor condition. This change applies throttling at individual condition instead of applying it generically at the subprocess level. The TaskMonitor condition will not be executed once per 10 seconds instead of one minute. This will allow for faster evaluation of worker idleness which would help in scaling down the worker fleet a lot more effectively.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

ashishgo-aws commented 1 month ago

These changes also need to be ported to 2.10.1.

I believe I have covered these in https://github.com/aws/amazon-mwaa-docker-images/pull/151. Is there anything you saw that is missing in that PR?

kuyperse commented 1 month ago

These changes also need to be ported to 2.10.1.

I believe I have covered these in #151. Is there anything you saw that is missing in that PR?

I didn't check the other PRs to see if the changes were ported there. Nothing seems to be missing. LGTM.