Model lifecycle management

TIBCOSoftware / flogo

Project Flogo is an open source ecosystem of opinionated event-driven capabilities to simplify building efficient & modern serverless functions, microservices & edge apps.

http://flogo.io

BSD 3-Clause "New" or "Revised" License

2.41k stars 288 forks source link

Model lifecycle management #397

Open mellistibco opened 5 years ago

mellistibco commented 5 years ago

Current behavior: When a Flogo app leverages the inference activity, the specified model cannot be changed without a restart of the process. The model must also be physically placed on the device, cloud storage, etc prior to the app being executed.

Expected behavior: Explore options to enable a more robust delivery of models and over the air update or models without requiring the engine to be restarted. Also consider support for champion/challenger model bakeoffs.

What is the motivation / use case for changing the behavior?

Enhance the usability in production scenarios

where it is either not possible or very challenging to gain access to the device, storage location, etc to update a model
where downtime is not an option to deliver new models

Additional information you deem important (e.g. I need this tomorrow):

abramvandergeest commented 5 years ago

Can I ask what you mean by "champion/challenger model bakeoffs"? Also, we can check timestamps of models to see if we need to reload? so we don't have to shutdown the engine.

mellistibco commented 5 years ago

@abramvandergeest thinking A/B testing. Basically load two models, split the inference load between models and eval the performance of each.

Also, kinda thinking we need some form of svc to push updated models to the running engine... I suppose we could also look at dropping models in a dir and the activity could just load the latest timestamp...

abramvandergeest commented 5 years ago

So I have been thinking of how to approach this. My thinking is that we can create an A/B testing activity that passes models/features/etc to the next activity (i.e. the inference activity). This activity would be built into the app in general with a config file set to only A, then we can add A/B options to the config file. I would suggest if we do this going with a pre-existing open source A/B testing framework. I don't know any off the top of my head but I would be surprised if one doesn't exist. Then we either use the svc or check dates to load the latest time stamp to actually change a model. If we do this in the context of the A/B test we would just change which output is used more. If that makes sense.

abramvandergeest commented 5 years ago

A google search provides this quora question on opensource A/B testing frameworks: https://www.quora.com/Which-are-the-best-open-source-A-B-testing-framework-to-use-in-your-projects