Open rawwar opened 1 month ago
cc @vincbeck @o-nikolas looks like a valid bug
Hi @rawwar i have tried the steps you have mentioned from my local, but not been able to re produce the issue, in my case its getting succeeded. is there any other information help to reproduce this issue? could you please provide.
First task:
Between there are couple of tasks triggered with seconds gap.
last task with after 4minutes.
@gopidesupavan and I connected last week to replicate the issue. But the problem is only happening on my local setup. I am testing this out on a fresh EC2 instance and will provide an update.
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
Apache Airflow version
main (development)
If "Other Airflow 2 version" selected, which one?
No response
What happened?
When invoking lambda functions using
LambdaInvokeFunctionOperator
, the task continues to run even after the actual lambda invocation is completed. It then throws aReadTimeoutError
.It is more common with Lambda functions that take more than 13 minutes to run. For Lambda functions that take more than 4 minutes, this is common when multiple tasks with
LambdaInvokeFunctionOperator
are triggered(I.e., Invoke the same lambda).I have followed the recommended settings as mentioned here:
My AWS connection extra parameter has the following json
I did set the max timeout to 15 minutes for the lambda function on AWS.
For the mentioned recommendations on the docs:
What you think should happen instead?
Tasks Should not run beyond the actual completion of the lambda invocations.
How to reproduce
Use the following DAG:
Create an AWS connection with the following json in the extra(You might need to add AWS
aws_session_token
andregion_name
to the extra:On AWS, create a lambda function, and update timeout to 15 minutes(That is the max possible value)
You can add
time.sleep(780)
(13 minutes) to your lambda code so that it runs for 13 minutes.Also decrease sleep time to 4 minutes and trigger the DAG multiple times quickly to reproduce ReadTimeout's
Operating System
ubuntu-22.04
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.27.0
Deployment
Docker-Compose
Deployment details
Docker file update as below
Docker-compose.yml
Anything else?
No response
Are you willing to submit PR?
Code of Conduct