search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.55k stars 14.15k forks source link

Deferred ECS tasks failing when reading logs from cloudwatch throws RequestTimeoutException #42521

Open pgieser opened 1 week ago

pgieser commented 1 week ago

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

8.27.0

Apache Airflow version

2.9.3

Operating System

Managed by Astronomer

Deployment

Astronomer

Deployment details

No response

What happened

Deferred ECS task failed because fetching logs from Cloudwatch timed out (AWS cloudwatch logs client threw RequestTimeoutException).

What you think should happen instead

Should probably be similar behavior as when the AWS cloudwatch logs client throws ResourceNotFoundException - it should be handled and return None instead of allowing the RequestTimeoutException to propagate.

See: https://github.com/apache/airflow/blob/84e8cdf67475c4b2eeadde99cb11eb02459cc9f5/airflow/providers/amazon/aws/triggers/ecs.py#L214

How to reproduce

Force the AWS cloudwatch logs client to throw a RequestTimeoutException when running an ECS task in deferred mode.

Anything else

No response

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 1 week ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.