astronomer / airflow-provider-great-expectations

Great Expectations Airflow operator
http://greatexpectations.io
Apache License 2.0
159 stars 54 forks source link

Feature request: Override connection properties #158

Open freefood89 opened 3 weeks ago

freefood89 commented 3 weeks ago

I am using this provider with a snowflake connection. Would it be possible to override a connection's database and warehouse (specific to snowflake)?

This is sort of similar to #74 and would be helpful since without it we have to create airflow connections for each warehouse/database we have with snowflake.

I could see how adding warehouse might be difficult since it's unique to snowflake, but I feel like database is doable. Maybe warehouse can be added by modifying the query param of the connection string ?

    gx_validate_snowflake = GreatExpectationsOperator(
        task_id="gx_validate_snowflake",
        conn_id='snowflake_sandbox',
        data_context_config=context.config,
        data_asset_name="TABLE",
        expectation_suite_name="suite_name",
        schema="SCHEMA",
        database="DATABASE",  # PROPOSED 
        conn_query={ "warehouse": "WAREHOUSE" } # PROPOSED 
    )

I'm willing to work on this and possible upstream changes if necessary.

freefood89 commented 1 week ago

actually #111 would address this issue