ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
Adds support when using PipelineController.add_step(). Currently, it's impossible to parse anything besides strings with _parse_step_ref() in parameter_override. This patch allows list, dics, and tuples to be recursively parsed for step reference substitution. Since recursive features can add computational complexity the new feature is hidden behind a flag that can be passed when adding a step, which makes this a non-breaking change.
Testing Instructions
If recursively_parse_parameters=True in PipelineController.add_step(), recursively parse parameters from parameter_override in lists, dicts, or tuples.
Examples:
parameter_override={'Args/input_file': ['${<step_name>.artifacts.<artifact_name>.url}', 'file2.txt']} will be correctly parsed.
parameter_override={'Args/input_file': ('${<step_name_1>.parameters.Args/input_file}', '${<step_name_2>.parameters.Args/input_file}')} will be correctly parsed.
Related Issue \ discussion
https://github.com/allegroai/clearml/issues/1089
Patch Description
Adds support when using
PipelineController.add_step(). Currently, it's impossible to parse anything besides strings with_parse_step_ref()inparameter_override. This patch allows list, dics, and tuples to be recursively parsed for step reference substitution. Since recursive features can add computational complexity the new feature is hidden behind a flag that can be passed when adding a step, which makes this a non-breaking change.Testing Instructions
If
recursively_parse_parameters=TrueinPipelineController.add_step(), recursively parse parameters from parameter_override in lists, dicts, or tuples. Examples:parameter_override={'Args/input_file': ['${<step_name>.artifacts.<artifact_name>.url}', 'file2.txt']}will be correctly parsed.parameter_override={'Args/input_file': ('${<step_name_1>.parameters.Args/input_file}', '${<step_name_2>.parameters.Args/input_file}')}will be correctly parsed.