Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
187 stars 44 forks source link

[FEATURE] Variables similar to Airflow that can be globally available #170

Open rebecca-rajshree opened 2 weeks ago

rebecca-rajshree commented 2 weeks ago

Is your feature request related to a problem? Please describe.

We have a usecase where we need to send a timestamp value from one workflow to another. In Airflow, it was achieved by setting it in the Variable and the other dag was able to access it. I couldn't find something similar to a global variable in databricks.

JobsParameters feature in brickflow allows us to pass values from one workflow to another by passing it via an operator like RunJobTask, JarTask, etc., But in cases where we don't want to use any of those operators, a global variable is really handy and helpful.

Cloud Information

Describe the solution you'd like If possible, please take it up with databricks to know if they would implement something like this in the future

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.