Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
187 stars 44 forks source link

[FEATURE] Set default schedule to "paused" for any Job in Test Env #75

Open jcnorman48 opened 10 months ago

jcnorman48 commented 10 months ago

Is your feature request related to a problem? Please describe. In order to minimize cost, any deployed job that is running on a schedule should have a state set as paused in the test environment.

Cloud Information

Describe the solution you'd like Any deployment of a Job/pipelines in brickflow should be set as "paused" if specified to be running on a schedule when deployed to any non prod environment.

Describe alternatives you've considered

Additional context

asingamaneni commented 9 months ago

We also need to probably need to hard set the schedule as None if the env is local.

asingamaneni commented 9 months ago

For local: we will default it to None, but will give an option to override and schedule it For dev, qa & prod: Folks should be able to decide the schedule in the workflow using ctx.env and conditional statements

pariksheet commented 3 months ago

For local: we will default it to None, but will give an option to override and schedule it For dev, qa & prod: Folks should be able to decide the schedule in the workflow using ctx.env and conditional statements

for local -- instead of setting schedule to None, we should make the workflow status PAUSE.

There are time, when team wants test Quartz expressions. PAUSED is better than override flag.

asingamaneni commented 3 months ago

I agree, makes sense!