StepFunctionStartExecutionOperator in MWAA does not throw error if IAM Role does not DescribeExecution permission

[sarkch]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

2.8.1

Apache Airflow version

2.8.1

Operating System

Managed Airflow

Deployment

Amazon (AWS) MWAA

Deployment details

No response

What happened

I am using StepFunctionStartExecutionOperator to execute a StateMachine.

StepFunctionStartExecutionOperator(
...
task_id="load_data",
deferrable=True,
waiter_delay=30,  # Poll for every 30 seconds
waiter_max_attempts=10,  # maximum number of attempts to poll for status
do_xcom_push=True,
)

if I look into the log of the task,

 {{waiter_with_logging.py:129}} INFO - Status of step function execution is: 
{{waiter_with_logging.py:129}} INFO - Status of step function execution is: 
{{waiter_with_logging.py:129}} INFO - Status of step function execution is: 

as you can see that airflow is not getting the current status (RUNNING, FAILED etc) of the StateMachine

What you think should happen instead

Expected Output when the State Machine is RUNNING

[2024-08-28, 17:01:06 UTC] {{waiter_with_logging.py:129}} INFO - Status of step function execution is: RUNNING
[2024-08-28, 17:02:06 UTC] {{waiter_with_logging.py:129}} INFO - Status of step function execution is: RUNNING

How to reproduce

So in my case I found the root cause of this problem. The IAM Role associated with the Airflow did not have the below permission

'states:DescribeExecution'

On the StateMachine execution arn.

arn:aws:states:<Region>:<accountId>:execution:<stateMachineName>:*

Before granting the permission

[2024-08-28, 01:41:53 UTC] {{waiter_with_logging.py:129}} INFO - Status of step function execution is:

After Granting the permission

[2024-08-28, 17:01:06 UTC] {{waiter_with_logging.py:129}} INFO - Status of step function execution is: RUNNING

Unfortunately, MWAA does not throw Accessdenied permission error if the IAM role does not have aforementioned permission on the execution arn of the StateMachine.

Anything else

No response

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[vincbeck]

I am not sure this is an Airflow issue but a MWAA issue

[sarkch]

Yes. this is more of an MWAA issue than Airflow. But in the questionnaire it asked which distribution of Airflow I am facing this problem and MWAA was one of the options. Hence, I thought the issues/bugs with MWAA are also tracked through here.

[vincbeck]

No, we are tracking here only issues related to Airflow. I'll close this one and re-open the other one you opened internally and will transfer it to MWAA team