allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.63k stars 653 forks source link

API calls fail for model with deleted parent task. #1299

Open bithoarder opened 2 months ago

bithoarder commented 2 months ago

Describe the bug

Using some python API calls (tested with settings tags) on models where the parent task have been deleted will fail with the error:

clearml.backend_interface.session.SendError: Action failed <400/16: models.edit/v1.0 (Invalid value for fields (expecting Task id): id=__DELETED__f5ed57f0511c47c59aa1fe706242722e, field=task)>

To reproduce

Archive an experiment task, then go to the archive list and delete the experiment task, but keep the model artifact. Then try to set the models tags from python: model.tags = ("test-tag")

Expected behaviour

The models tags should be updated.

Environment

Workaround

Reaching into the model internals and clearing the task id before calling model.tags will allow the API call to complete.

if model.task.startswith('__DELETED__'):
    model._get_base_model().data.task = None
ainoam commented 2 months ago

Thanks for reporting @bithoarder - we'll take a look.