search

astronomer / astronomer-providers

Airflow Providers containing Deferrable Operators & Sensors from Astronomer
https://astronomer-providers.rtfd.io/
Apache License 2.0
137 stars 25 forks source link

Test aws_nuke dag with multiple worker on celery executor #1045

Closed utkarsharma2 closed 10 months ago

utkarsharma2 commented 1 year ago

Is your feature request related to a problem? Please describe. We have observed failures for the aws_nuke command, we suspect it might be because of the execution of credentials-related changes in one operator and the aws_nuke command on a different operator. Since the celery executor only has a single worker it's working but it might fail if we have more workers. So the credentials-related operation executes in one worker node and the aws_nuke executes in a different worker.

ref: https://github.com/astronomer/astronomer-providers/blob/0f8635742f9171427ea482945fddcbdc1372a604/astronomer/providers/amazon/aws/example_dags/example_aws_nuke.py

Describe the solution you'd like We want to test this theory.

vatsrahul1001 commented 10 months ago

Already been tested and verified. Closing this