This is an enhancement to account for below use cases
passing sql file instead of query string ( which would be an additional hop everytime we call operator)
passing in custom sql to extract data from Unity catalog. Current code only accepts guard rail sql filters against select statements. using this enhancement user has better control on the source
How Has This Been Tested?
Tested the updated operators for multiple scenarios
using the sql file
using query string
using both (negative testing)
Tested the UcToSnowflakeOperator for below scenarios
Incremental
Full load with dbx_sql and tables
write mode
Screenshots (if appropriate):
Make check result
Updated brickflow operators workflow
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
Update Snowflake and UctoSnowflakeOperator Operators to accept sql file and custom db_sql queries as inputs
Description
It is an enhancement to pass sql_file to Snowflake Operator and custom sql to extract data from Unity catalog.
Below are the changes implemented as part of this PR
Related Issue
https://github.com/Nike-Inc/brickflow/issues/115
Motivation and Context
This is an enhancement to account for below use cases
How Has This Been Tested?
Tested the updated operators for multiple scenarios
Screenshots (if appropriate):
Make check result
Updated brickflow operators workflow
![image](https://github.com/Nike-Inc/brickflow/assets/43893120/8916cef3-92d7-4fbe-9259-705d7a3e988a)
Types of changes
Checklist: