search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.69k stars 655 forks source link

Modifying task name under "EXPERIMENTS" in the UI does not reflect in the model checkpoint's name under "MODELS" #1179

Open nizamreplica opened 11 months ago

nizamreplica commented 11 months ago

Describe the bug

If one were to change the task name under experiments during a run, the change is not reflected in the model name prefix when checkpoints are generated under models Screenshot 2023-12-22 at 2 02 07 pm

This would make sense if the task has aborted or completed, but should work when it is running.

Note that the image attachments for the sake of this issue description are on status=ABORTED but the bug was found on status=RUNNING

To reproduce

1) Start a task that saves something as model checkpoints under models 2) While it's still running edit the task name in ClearML UI Screenshot 2023-12-22 at 7 18 32 pm 3) Now instead of model checkpoints being saved with prefix test_base_model_3, it still saves them as test_base_model_2

Expected behaviour

The model checkpoints should be saved with prefix test_base_model_3

Environment

Server type (self hosted) ClearML SDK Version: 1.13.2 ClearML Server Version: 1.12.1-397 Python Version: 3.10.13 OS (Linux, Ubuntu)

ainoam commented 10 months ago

@nizamreplica Indeed, ClearML does not provide UI control of tasks while they are executed. You can only control a tasks configuration pre-execution ('draft') and modify descriptive elements (e.g. name, tags, description etc.) post execution.

Manually changing the tasks behaviour as it executes could lead to incoherent results, and prevent reproducibility and so is not supported. Does this make sense?