Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 36 forks source link

[FEATURE] Add option to pass Sql file to the Snowflake Operator and UctoSnowflake Operator #115

Closed bsangars closed 2 months ago

bsangars commented 3 months ago

Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is.

  1. We don't have option to pass the sql file as a parameter to the Snowflake Operator.
  2. We don't have an option to pass custom sql/ file to extract from Unity Catalog in UcToSnowflakeOperator.

Cloud Information

Describe the solution you'd like

  1. Write a block of code to read sql file and convert it to sql query as a string and execute it against snowflake connection
  2. Handle the scenario to use custom sql query or file to extract data from UC.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.