ECS Executor with EC2 launch type doesn't work as platform version cannot be unset which is a requirement

[AdamStrLSEC]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.25.0

Apache Airflow version

2.9.3

Operating System

Debian GNU/Linux 12

Deployment

Other Docker-based deployment

Deployment details

Using the upstream Airflow docker images customised using by exporting the following env vars:

AIRFLOWCOREEXECUTOR=airflow.providers.amazon.aws.executors.ecs.ecs_executor.AwsEcsExecutor AIRFLOWAWS_ECS_EXECUTORCLUSTER= AIRFLOWAWS_ECS_EXECUTORCONTAINER_NAME= AIRFLOWAWS_ECS_EXECUTORTASK_DEFINITION= AIRFLOWAWS_ECS_EXECUTORLAUNCH_TYPE=EC2 AIRFLOWAWS_ECS_EXECUTORSECURITY_GROUPS= AIRFLOWAWS_ECS_EXECUTORSUBNETS= AIRFLOWAWS_ECS_EXECUTORREGION_NAME=eu-west-2

What happened

Currently when specifying the launch type for the ECS executor as "EC2" the current default being set for platform_version is being set to "Latest" which is incompatible for the EC2 launch type. When you don't export the AIRFLOWAWS_ECS_EXECUTORPLATFORM_VERSION the current upstream airflow docker image will still put platform_version=latest into the airflow.cfg which will cause the ECS executor to fail as the AWS API says that the platform version parameter cannot be specified when an EC2 launch type is used when attempting to run a DAG with this executor.

If you specify an empty string for AIRFLOWAWS_ECS_EXECUTORPLATFORM_VERSION this stops the airflow.cfg being populated with a value for platform_version but the executor will still pass a platform_version to the resulting RunTask API call at which point AWS returns saying that the platform_version cannot be blank. Setting any other value such as null or anything else results in the original error of the platform version parameter cannot be specified when using EC2 as the launch type

What you think should happen instead

The ECS Executor should verify that if EC2 is used as the launch type that the platform_version parameter is not passed to the API call. It seems the ECS Operator had a similar issue that was fixed as part of https://github.com/apache/airflow/issues/17276

How to reproduce

Setup an ECS Cluster based on EC2 as the launch type with a task definition and anything else required as part of Airflow Export the above variables into the docker image and run the webserver and scheduler and attempt to schedule a DAG run.

Anything else

No response

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[eladkal]

cc @o-nikolas

[ferruzzi]

Thanks for calling this out! We'll look into it, but with Airflow Summit coming next week it may have to wait till after that.