Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 39 forks source link

Add ability to set timeout health check in workflows #58

Closed brent-johnson closed 10 months ago

brent-johnson commented 10 months ago

Enable health parameters to be passed into workflow and task to set timeout warning notifications

Description added health option to Workflow class

How Has This Been Tested? tests updated to cover new functional dev test by deploying workflows to databricks with health option

Screenshots (if appropriate): Types of changes

asingamaneni commented 10 months ago

@brent-johnson looks like there are linting issues. Can you please run make fmt in your local and the push the changes. To verify locally you can run make test

brent-johnson commented 10 months ago

@asingamaneni Yep. Sorry about that. I'm not normally a python developer. I've added a commit that fixes the unit tests and formatting.

codecov[bot] commented 10 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (b2af63d) 88.48% compared to head (1e6cfd3) 88.49%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #58 +/- ## ======================================= Coverage 88.48% 88.49% ======================================= Files 22 22 Lines 3187 3189 +2 ======================================= + Hits 2820 2822 +2 Misses 367 367 ``` | [Files](https://app.codecov.io/gh/Nike-Inc/brickflow/pull/58?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Nike-Inc) | Coverage Δ | | |---|---|---| | [brickflow/codegen/databricks\_bundle.py](https://app.codecov.io/gh/Nike-Inc/brickflow/pull/58?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Nike-Inc#diff-YnJpY2tmbG93L2NvZGVnZW4vZGF0YWJyaWNrc19idW5kbGUucHk=) | `90.03% <ø> (ø)` | | | [brickflow/engine/task.py](https://app.codecov.io/gh/Nike-Inc/brickflow/pull/58?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Nike-Inc#diff-YnJpY2tmbG93L2VuZ2luZS90YXNrLnB5) | `94.92% <100.00%> (+0.01%)` | :arrow_up: | | [brickflow/engine/workflow.py](https://app.codecov.io/gh/Nike-Inc/brickflow/pull/58?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Nike-Inc#diff-YnJpY2tmbG93L2VuZ2luZS93b3JrZmxvdy5weQ==) | `100.00% <100.00%> (ø)` | |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

asingamaneni commented 10 months ago

@asingamaneni Yep. Sorry about that. I'm not normally a python developer. I've added a commit that fixes the unit tests and formatting.

Thanks much for creating the pr and making the changes.