MLOps in Python

[andrewcboardman]

Context and Problem Statement

In ML projects, we need to track datasets, models and experiments. MLOps is the process of tracking experiments and moving machine learning models into production systems in a controlled way. MLOps systems can also be used to ease the production of reports on ML experiments.

Decision Drivers

• Convenience of use
• Avoiding initial setup work that slows down a project
• Producing professional-looking updates on experiments and work performed
• Avoiding dependencies (particularly non-open source dependencies) which may be unattractive to clients.

Considered Options

• Custom scripts (this can be attractive initially and for quick experiments but gets cumbersome over time)
• Weights and Biases (this requires a license so is unattractive if the product will be handed over to clients for maintenance)
• ClearML (will experiment with this)

[andrewcboardman]

Notes on ClearML:

• Writes a variety of logs to a ClearML server. We would want to run our own server, I think.
• ClearML can be set to save models to a server without obvious warnings. We would definitely not want to do this as they can be considered PID. I would therefore be somewhat cautious about integrating ClearML into my code until I fully understand its functionality.