kubeflow / code-intelligence

ML-Powered Developer Tools, using Kubeflow
https://medium.com/kubeflow/reducing-maintainer-toil-on-kubeflow-with-github-actions-and-machine-learning-f8568374daa1?source=friends_link&sk=ac77444f00c230e7d787edbfb0081918
MIT License
55 stars 21 forks source link

[label bot] Continuously train and deploy label bot model - using GitOps #155

Open jlewi opened 4 years ago

jlewi commented 4 years ago

We'd like to periodically (weekly) retrain the model and deploy the latest model to production.

This would allow us to benefit from

We'd like to use GitOps. We should automatically create a PR to update the model and once that model is approved and merged the new model should get deployed automatically.

There are three pieces

  1. Automating training
    • Currently this means automating running the notebook that trains an AutoML model
  2. Automatically creating a PR to update the model config to use the latest AutoML model
  3. Use GitOps tools to automatically sync the latest model configs down to the cluster

The last step was taken care of by #152

For step 2 here's what I'm thinking create a tekton task that does the following

One thing I'm not sure about yet is how to perform the above in a reconcile loop; i.e. only trigger the above if the config isn't already pointing at the latest model

jlewi commented 4 years ago

I created a GitHub app to be used to create PRs https://github.com/organizations/kubeflow/settings/apps/label-bot-cd

Created a private key Finger print a0:3c:a9:87:e9:46:df:55:71:92:87:a1:bc:71:d6:6b:e0:7a:2f:89

Private key is stored in secret manager https://cloud.console.google.com/security/secret-manager/secret/label-bot-cd-github-app-pem?project=issue-label-bot-dev

Installed it on the kubeflow/code-intelligence repo

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the labels:

Label Probability
kind/feature 0.95

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

jlewi commented 4 years ago

controller is working locally; Here's an auto-created PR. https://github.com/kubeflow/code-intelligence/pull/166

Now I just need to get it deployed in the cluster.

jlewi commented 4 years ago

Its deployed in production. Here's an auto created PR https://github.com/kubeflow/code-intelligence/pull/173

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the labels:

Label Probability
area/jupyter 0.77

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.