search

astronomer / airflow-provider-great-expectations

Great Expectations Airflow operator
http://greatexpectations.io
Apache License 2.0
157 stars 53 forks source link

Use Snowflake provider to build connection string #98

Closed ivanstillfront closed 1 year ago

ivanstillfront commented 1 year ago

This change uses the SnowflakeHook from the airflow.providers.snowflake.hooks.snowflake module to build the connection string.

Great Expectations expects a different configuration structure for a datasource that uses Snowflake and private key authentication. PR #93 does not cover all possible ways a Snowflake connection can be configured in Airflow.

I thought it would be best to optionally rely on the Snowflake provider package and fall back to the existing functionality if the provider package is not available (tho I doubt anyone would use Snowflake without the provider package)

Instead of using the SnowflakeHook, I could duplicate the behaviour from the hook into the GE Operator? It is up to the maintainer imho.

Please let me know if you have any questions/concerns or would like me to change this in any way.

ivanstillfront commented 1 year ago

Really appreciate this PR, did have a few specific questions here I'd like addressed before merging. Test coverage looks good, have you tried this in a dev environment?

Thank you for your feedback @denimalpaca . Yes this has been running on our development environments since I opened this PR.