Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 39 forks source link

adding uc to snowflake operator #69

Closed ckonuganti closed 9 months ago

ckonuganti commented 9 months ago

Adding a new plugin to copy data from UC to snowflake and run snowflake queries.

Description

Added two plugins/operators to run snowflake queries and copy data from unity catalog to snowflake. these plugins helps us run the workloads from one environment instead of switching between dbx and sf. As a future enhancement , want to include support for more credential stores #70

Related Issue

63

Motivation and Context

As Enterprises might be using multiple solutions for data warehousing . we have a need to copy data from Unity Catalog to snowflake to support few scenarios. Instead of every team handling this common process in their own unique way, creating this plugin that can copy from UC table to Snowflake Table (Full Load, Incremental) and run queries directly on snowflake.

How Has This Been Tested?

We tested the code changes manually and documented(will attach screenshots). creating a new issue for automating the test cases #68.

Screenshots (if appropriate):

Screenshot 2023-12-08 at 10 14 35 AM

Types of changes

Checklist:

codecov[bot] commented 9 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (fcf1b1c) 88.49% compared to head (78edd7f) 93.33%. Report is 1 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #69 +/- ## ========================================== + Coverage 88.49% 93.33% +4.84% ========================================== Files 22 22 Lines 3189 4919 +1730 ========================================== + Hits 2822 4591 +1769 + Misses 367 328 -39 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.