Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 36 forks source link

added fix for snowflake operator to accept authenticator and make parameters optional based on load_type #92

Closed ckonuganti closed 6 months ago

ckonuganti commented 6 months ago

Description

Added code to handle authenticator issue posted in issue #85 Added snowflake library installation to avoid initialization failures for brickflow plugins enhanced the code to accept parameters based on load_type instead of making every parameter mandatory.

Related Issue

fix #85

Motivation and Context

At this point code is failing for user id's which require extra authentication mechanism which is blocking multiple brickflow consumers. As we added snowflake operator to initialize with brickflow_plugins it is looking for snowflake library and causing issues to some customers, fixing this issue. Current version of the operator expects optional parameters like sf_cluster_keys, incremental_filter, sf_grantee_roles even when they are not required for customers and causing issues.

How Has This Been Tested?

Created a wheel file , workflow and tested with user id that require and doesn't require extra authentication. ran multiple test cases with different combination of input params. created a notebook with operator code and ran multiple manual unit tests.

Screenshots (if appropriate):

Types of changes

Checklist:

asingamaneni commented 6 months ago

@ckonuganti can you please post some screenshots of this working

ckonuganti commented 6 months ago

all_test_cases_uc_2_sf_test Ad task_with_libraries_uc_2_sf_testing ding workflow screenshots with test cases.