Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 39 forks source link

[FEATURE] Snowflake operator to load data from UC #63

Open ckonuganti opened 10 months ago

ckonuganti commented 10 months ago

Is your feature request related to a problem? Please describe. As Enterprises might be using multiple solutions for data warehousing . we have a need to copy data from Unity Catalog to snowflake to support few usecases. Instead of every team handling this common process in their own unique way, create a plugin that can copy from UC table to Snowflake Table (Full Load, Incremental).

Cloud Information

Describe the solution you'd like Similar to an operator in airflow, create plugin(s) in brickflow to support dataload from UC to Snowflake.

Describe alternatives you've considered Alternatives are creating own scripts by each team {member] and move data using spark task.

Additional context