Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 36 forks source link

[FEATURE] Santise input of `brickflow_internal_only_run_tasks` parameter #105

Closed maxim-mityutko closed 4 months ago

maxim-mityutko commented 5 months ago

Is your feature request related to a problem? Please describe. This parameter expects coma delimited list of tasks that should be executed. I noticed that on multiple occasions engineers provide this list with (a) trailing space after coma, e.g task_1, task_2; (b) in python list form, e.g. ['task_1', 'task_2']

Cloud Information

Describe the solution you'd like Cleanup the input while parsing the contents of brickflow_internal_only_run_tasks parameter:

Feels like https://github.com/Nike-Inc/brickflow/blob/b07ebfb88517d03dc0191cf587a84b7e6e54b82d/brickflow/engine/task.py#L677-L690 is the logical place for this.

Describe alternatives you've considered Educating users is always an option.

Additional context Add any other context or screenshots about the feature request here.

asingamaneni commented 4 months ago

This is a good idea and needed.