search

openclimatefix / satflow

Satellite Optical Flow with machine learning models
https://satflow.readthedocs.io/en/stable/
MIT License
61 stars 10 forks source link

Add CI/CD Model Runs #103

Open jacobbieker opened 3 years ago

jacobbieker commented 3 years ago

Detailed Description

To know how changes are affecting model performance, it could be nice to be able to run a model with commands from PRs and record how it compares to the baseline/other models

GitHub has some actions that can be accomplished to do that: https://github.blog/2020-06-17-using-github-actions-for-mlops-data-science/

Context

We need to compare our models to other PV predictions, and so this could be a nice way to record some model performance, in addition to Neptune.ai

Possible Implementation

https://github.blog/2020-06-17-using-github-actions-for-mlops-data-science/ https://github.com/machine-learning-apps/actions-ml-cicd

dudeperf3ct commented 2 years ago

@jacobbieker I would be interested in submitting a PR!

Do we have a workflow in the works? e.g. from above blog would be using chatops gh action to communicate using pr, some gh action to submit (Argo or CML) ML workflow (we have to select which models to run against by modifying config.yaml or some kind of matrix) in addition to logging to Neptune and display the metrics and plots on the pr for ease of comparison.

Some questions like do we trigger this workflow by making some changes to default parameters or simply run with default parameters for selected models for every trigger?

jacobbieker commented 2 years ago

Hi! We currently don't have a workflow for this, and there isn't one in the works. I think the ideal way would be to trigger it based off when a model or datamodule config changes. We've used CML before for Satip.