aws-samples / sagemaker-custom-project-templates

MIT No Attribution
214 stars 156 forks source link

Fixes #15 - Only approved models trigger the deployment pipeline #16

Open zoltan-fedor opened 2 years ago

zoltan-fedor commented 2 years ago

This fixes https://github.com/aws-samples/sagemaker-custom-project-templates/issues/15.

Based on the design of mlops-template-gitlab, the model deployment pipeline should only be triggered for 'Approved' models.

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

kirit93 commented 2 years ago

Will this design ensure the CI/CD pipeline for deployment will be triggered when models are moved from "Approved" to "Rejected"? The 1P templates roll-back the endpoint to the latest approved model if the deployed model's status changes to "Rejected".

zoltan-fedor commented 2 years ago

The CI/CD pipeline only gets triggered when the model moves to "Approved". Once it is "Approved" the pipeline is already triggered. You can't "untrigger" it by moving it to "Rejected" from "Approved". A model being in "Pending" should either go to "Approved" or "Rejected" and not first to "Approved" and then to "Rejected". I don't really see why would you approve a model first just to reject it later.

If you have a valid use-case to have models move from "Approved" to "Rejected", then you will need to extend the design to re-trigger an event to trigger the CI/CD when a model gets rejected and so the latest approved model re-triggers the deployment pipeline.

kirit93 commented 2 years ago

What if you "Approve" a model, it's performance starts to drift, and you need to roll back to a previous version of the model? Changing the state to "Rejected" should allow you to trigger an automated rollback.

zoltan-fedor commented 2 years ago

Sure, feel free to make a PR for that. What is certain, that a non-approved model should NOT proceed to deployment - which is what this PR fixes.