search

Azure / mlops-v2

Azure MLOps (v2) solution accelerators. Enterprise ready templates to deploy your machine learning models on the Azure Platform.
https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment
MIT License
475 stars 224 forks source link

Model Promotion #75

Closed levalencia closed 1 year ago

levalencia commented 1 year ago

I wasnt able to find in the code or in the documentation anything related to model promotion and data drift.

If I have a daily ingestion of new data and I run the training pipeline on a daily basis, probably at some point the daily model will have better metrics that the one deployed.

How can I replace the model ONLY if the metrics are better? Could you provide some guidance or links if it exists?

setuc commented 1 year ago

@levalencia One approach that we have is code based. This will only register the model if the pre-defined metrics are better

https://github.com/Azure/mlops-project-template/blob/5801db4951eab35d0f443c952314d7de60c6da11/classical/aml-cli-v2/data-science/src/evaluate.py#L132

This is one such approach. This has the limitation that if you change your mind about how the models are evaluated, then you will have to retrain the models.

An alternate approach that you can try is PR based approach aka Trunk based MLOps, where you keep your development model on a different branch and your in-production model on the main branch. Then you can compare the metrics between the two, to see if the model can be promoted to the main branch, i.e. production.

As far as data drift goes, you can work with some development that a few of my colleagues had done: https://github.com/Azure/data-model-drift