astronomer / airflow-provider-great-expectations

Great Expectations Airflow operator
http://greatexpectations.io
Apache License 2.0
159 stars 54 forks source link

Snowflake connection failure when only using the Airflow UI fields to set the connection #83

Closed TJaniF closed 1 year ago

TJaniF commented 1 year ago

Hi!

I noticed that when only using the Airflow UI fields to set a connection the Extra JSON rendered has different keys than what the GXO is looking for.

Extra field automatically rendered from connection UI fields: {"account": "myacc", "warehouse": "HUMANS", "database": "DWH_DEV", "region": "us-east-1", "role": "myrole", "insecure_mode": false}

The GXO is looking for extra__snowflake__account, extra__snowflake__region etc. I think this might have been how keys used to be rendered? Setting the connection with these keys adding extra__snowflake__ in front of the parameters everything works as expected.

Error trace:


[2023-01-05, 13:37:40 UTC] {great_expectations.py:454} INFO - Instantiating Data Context...
[2023-01-05, 13:37:40 UTC] {base.py:73} INFO - Using connection ID 'galaxy_snowflake_etl' for task execution.
[2023-01-05, 13:37:40 UTC] {taskinstance.py:1772} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/great_expectations_provider/operators/great_expectations.py", line 456, in execute
    self.build_runtime_datasources()
  File "/usr/local/lib/python3.9/site-packages/great_expectations_provider/operators/great_expectations.py", line 366, in build_runtime_datasources
    self.datasource = self.build_configured_sql_datasource_config_from_conn_id()
  File "/usr/local/lib/python3.9/site-packages/great_expectations_provider/operators/great_expectations.py", line 259, in build_configured_sql_datasource_config_from_conn_id
    conn_str = self.make_connection_string()
  File "/usr/local/lib/python3.9/site-packages/great_expectations_provider/operators/great_expectations.py", line 246, in make_connection_string
    uri_string = f"snowflake://{self.conn.login}:{self.conn.password}@{self.conn.extra_dejson['extra__snowflake__account']}.{self.conn.extra_dejson['extra__snowflake__region']}/{self.conn.extra_dejson['extra__snowflake__database']}/{schema}?warehouse={self.conn.extra_dejson['extra__snowflake__warehouse']}&role={self.conn.extra_dejson['extra__snowflake__role']}"  # noqa
KeyError: 'extra__snowflake__account'```
denimalpaca commented 1 year ago

Yes, looks like this was a change with version 4.0.0: Update snowflake hook to not use extra prefix (#26764)

Would be greatly appreciated if you submitted a PR! Should be an easy fix 😄