search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.66k stars 651 forks source link

Ensemble models #318

Open MFreidank opened 3 years ago

MFreidank commented 3 years ago

Hello there,

I was wondering what the best way to manage an ensemble of models through clearML is? There seem to be assumptions that a single task maps to a single output model, see #224.

I'd like to understand the best way to:

  1. store/track checkpoints for an ensemble of models grouped together
  2. version/document an ensemble of models together in the registry
  3. deploy this ensemble of models, ideally as a unit that can be queried together

Could you please help me understand how this could be done?

Many thanks in advance and kind regards, MFreidank

bmartinn commented 3 years ago

Hi @MFreidank

Good point and you are in luck, the next ClearML version will support multi-models 🎉 (finally) Basically instead of single model, a Task will be able to hold multiple models in a key/value alike structure, where the key is the model filename (or manually set model name)

Until 0.18 is out (hopefully towards the end of the month), here is what you can do: Currently multi models are stored in the ClearML artifactory and each one has a unique ID and a link back to the creating Task. The missing piece if for the Task to link back to multiple models, based on their names (currently it links back to the last one). That said, if you do Task.get_task('task_id_here').models['output'] you will get a list of all the output model objects, see docs and here. With the Model object you can query/set the name/tags/url etc. of the model itself.

Specifically regrading your questions:

  1. As explained above you can access all the models a Task produced by querying the Task (or querying the models for the creating Task ID), then you have a list of models you can deploy. The question becomes more complicated if we have multiple snapshots of these models created by the same Task, if this is the case the list is order by the order of the model storage so you could pick the last 3 (let's assume we have 3 models in the ensemble) from the list (you can also verify based on the model name that these are 3 different models and not the same model in different snapshots)
  2. I would add tags to the "chosen" models, so that you know they are part of a package. Obviously the question is how you would know, maybe as part of the training code?! Or a quality-control process testing all models on a blind dataset and choosing the best performing ensemble set ?
  3. I guess that depends on how you serve models and where you store them (BTW: serving integration will also be part of the next release ;)

What do think ?