allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.69k stars 655 forks source link

PipelineDecorator components not updating args when parent pipeline is cloned with new args #1092

Closed NicholasCJL closed 1 year ago

NicholasCJL commented 1 year ago

Describe the bug

Cloning a pipeline created with PipelineDecorator.pipeline with pipeline steps created with PipelineDecorator.component causes the pipeline steps to retain their original arguments, even when the pipeline calling them has updated arguments.

To reproduce

Minimal code to reproduce:

from clearml.automation.controller import PipelineDecorator

@PipelineDecorator.component(execution_queue="default")
def print_func(arg_1, arg_2):
    print(arg_1)
    print(arg_2)

@PipelineDecorator.pipeline(name="pipeline_bug", project="bug")
def bugged_pipeline(arg_1, arg_2):
    print_func(arg_1, arg_2)
  1. Call the pipeline function with some arguments (e.g. arg_1="1", arg_2="2") and execute it remotely.
  2. Upon completion of the pipeline, make a clone of the pipeline in the experiments page.
  3. Modify the arguments for the cloned pipeline (e.g. arg_1="3", arg_2="4").
  4. Run the cloned and modified pipeline.

Expected behaviour

The pipeline step print_func in the cloned pipeline should receive the new pipeline args arg_1 and arg_2. Instead, the pipeline step retains its original arguments from the first time it was called in the original pipeline.

Note: The same behaviour is also observed if the original pipeline gets run again with modified arguments without cloning.

Environment

AlexandruBurlacu commented 1 year ago

Hey @NicholasCJL, we fixed this issue in our most recent release, ClearML SDK 1.12.2. Can you try it and let us know if that fixes your problem?

NicholasCJL commented 1 year ago

Confirmed that it's fixed on 1.12.2. Thanks @AlexandruBurlacu!