astronomer / astro-sdk

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
https://astro-sdk-python.rtfd.io/
Apache License 2.0
329 stars 40 forks source link

output_table control in "aql.transform" #2082

Open Andrew-Wichmann opened 6 months ago

Andrew-Wichmann commented 6 months ago

Please describe the feature you'd like to see I would really like to be able to configure the output_table for the TransformOperator when using the @transform decorator.

Describe the solution you'd like

PRODUCTION_DATABASE = Table(
    name="IMAGE",
    conn_id=SNOWFLAKE_CONN_ID,
)

@dag(
    schedule="@daily",
    start_date=datetime(2023, 12, 12),
)
def fetch_missing_files():
    @aql.transform(output_table=Table(conn_id="some_other_db"))
    def fetch_image(prod_database):
        return """
        SELECT image
        FROM {{prod_database}}
        WHERE <condition>
        """
    fetch_image(PRODUCTION_DATABASE)

Are there any alternatives to this feature? :thinking: I'm not sure. I mean, I could just not use the decorator and initialize the Operator myself, but that's not as fun.

Additional context Nope. I love the product though. I'm really excited to start using more Airflow =)

Acceptance Criteria