Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 39 forks source link

Fix issue 48 and 60 related to the WorkflowDependencySensor #61

Closed BrendBraeckmans closed 10 months ago

BrendBraeckmans commented 10 months ago

Description

This PR contains following changes

Related Issue

Motivation and Context

At this point in time the WorkflowDependencySensor doesn't work at all which blocks multiple users of brickflow. I know that in the future this functionality will not be needed anymore in brickflow as it's become Databricks native so can be achieved through asset bundles. This however needs development at brickflow side so this PR fixes the issues we face till this is done.

How Has This Been Tested?

I build my own wheel based on this PR and tested it extensively on a workflow making use of the WorkflowDependencySensor

Screenshots (if appropriate):

Types of changes

Due to the change in how the token gets handled in WorkflowDependencySensor users will need to adapt their code when they upgrade. I however think there are no users of this feature at the moment as it's not working right now.

Checklist:

codecov[bot] commented 10 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (5a10093) 88.44% compared to head (196db3b) 88.49%. Report is 5 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #61 +/- ## ========================================== + Coverage 88.44% 88.49% +0.04% ========================================== Files 22 22 Lines 3176 3189 +13 ========================================== + Hits 2809 2822 +13 Misses 367 367 ``` [see 5 files with indirect coverage changes](https://app.codecov.io/gh/Nike-Inc/brickflow/pull/61/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Nike-Inc)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.