search

aws / amazon-mwaa-docker-images

Apache License 2.0
24 stars 11 forks source link

Porting auto-scaling internal code #84

Closed rafidka closed 3 months ago

rafidka commented 3 months ago

Issue #, if available: #46

Description of changes: This PR ports the internal code we have related to auto-scaling. It ensures graceful shutdown of workers in case of auto-scaling.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

rafidka commented 3 months ago

@ashishgo-aws

Overall, the change looks fine. Two things though:

1. Where is this logic placed? https://code.amazon.com/packages/AWSPinwheelImages/blobs/66f9433e722b91e4d01a517c13b5ba23b874dcc2/--/images/airflow/2.8.1/logs/console_handler.py#L123-L141

Ah, I missed this part. 

2. What testing has been done so far related to the autoscaling changes?

This is still WIP, so testing is still on-going. I appreciate if you share some testing tips/DAGs that can help with this.

rafidka commented 3 months ago

@ashishgo-aws , I addressed all comments and updated the PR. Please take a look. Thanks!