search

Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 39 forks source link

[FEATURE] Add support for run time parameters while deploying workflows #65

Open pariksheet opened 9 months ago

pariksheet commented 9 months ago

Currently user can specify custom params as hardcoded value

wf = Workflow(
    "my-workflow",
    default_cluster=cluster,
    common_task_parameters={'my_key': 'my_value'}
)

And there is no way to pass params at deployment time poetry run brickflow projects deploy -e local

Cloud Information

Describe the solution you'd like User will pass the params (max 10) at deployment time poetry run brickflow projects deploy -e local -p key1=value1 -p key2=value2

These params will be available as string in task parameters as brickflow_runtime_params.

For above example, brickflow_runtime_params='{'key1': 'value1', 'key2': 'value2'}'

Describe alternatives you've considered User can get it working using ENVIRONMENT variable, not very ideal.

Additional context Add any other context or screenshots about the feature request here.

asingamaneni commented 7 months ago

Disccussed with @pariksheet and @stikkireddy - We will be going with BRICKFLOW_PROJECT_PARAMS and use cli to pass the params. The parameters would be available through ctx and a helper function.