EcsRunTaskOperator should cast rendered networkConfiguration template field

[mmyers5]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.16.0

Apache Airflow version

2.8.1

Operating System

N/A

Deployment

Amazon (AWS) MWAA

Deployment details

N/A

What happened

Hi! I'm submitting this issue about the EcsRunTaskOperator and its templated fields, which are difficult to work with at my current job. Multiple data engineers have tried to find an elegant workaround, but I think we're at wits' end.

Passing in a templated string to the EcsRunTaskOperator network_configuration argument, whether using '{{ var.json... }}' or '{{ var.value... }}', passes it as a string which fails parameter validation:

(Parameter validation failed:
Invalid type for parameter networkConfiguration, value: { "awsvpcConfiguration": { "securityGroups": [ "sg-***" ], "subnets": [ "subnet-***", "subnet-***", "subnet-***" ] } }, type: <class 'str'>, valid types: <class 'dict'>; 3150)

We've been able to get around this by:

1. Using Variable.get(...) in the root code, which is not ideal since we have a secrets backend that charges us for every call, which is made whenever a DAG is compiled. I think we'd have to create a new task in every DAG to properly render this variable in XCOM, which is unsustainable and clutters our DAGs.
2. Setting render_template_as_native_obj=True at the DAG level, which is not ideal since EcsRunTaskOperator also renders overrides, which we use to pass environment variables to our containers. Any variable that could be inferred as an int is automatically cast to int, which fails parameter validation. I think we'd have to update our variables in the secrets backend to explicitly render the strings, which is not ideal.

What you think should happen instead

I think EcsRunTaskOperator should try to cast any string input to a dict object. Otherwise, the template fields are fairly combersome to use.

How to reproduce

Create an EcsRunTaskOperator and pass in a template string as a parameter. For example:

from datetime import datetime

from airflow import DAG
from airflow.providers.amazon.aws.operators.ecs import EcsRunTaskOperator

from utils import default_args, dag_callback, datadog_ecs_container

with DAG(
    dag_id='test_dag',
    schedule=None,
    start_date=datetime(2024, 10, 4),
    catchup=False,
    tags=[ENVIRONMENT],
    render_template_as_native_obj=False
) as dag:
    test_task = EcsRunTaskOperator(
        task_id = 'test_task',
        cluster='my-cluster',
        task_definition='my-task-definition',
        launch_type='FARGATE',
        overrides={
            'containerOverrides': [
                {
                    'name': 'my-container',
                    'environment': [
                        {
                            'name': 'ENVIRONMENT_VARIABLE_A',
                            'value': 'environment_variable_a_string_value'
                        },
                        {
                            'name': 'ENVIRONMENT_VARIABLE_B',
                            'value': '123456789'
                        }
                    ]
                }
            ]
        },
        network_configuration='{{ var.value.ecs_network_configuration_json }}'
    )

    test_task

Anything else

No response

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[jayceslesar]

One fix could be making a cast here (or some private method to correctly cast the overrides before they are passed to the run_task call) https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/operators/ecs.py#L461