search

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
9.04k stars 1.89k forks source link

Restore trainer code from model.zip #3673

Closed sergey-tihon closed 5 years ago

sergey-tihon commented 5 years ago

How can I restore the code that was used to create model.zip? Is model archive is self-containing?

Use case:

Let's say I have 10 models in production, created by few engineers. After some time (1 year) I find out that one model does not perform as good as it was before and I want to retrain it using the same algorithms but with more training data.

Should I maintain all version of trainers (code & params) and their relationships with generated model files separately or I will be able to restore code in the future?

Something like

mlnet gen-proj "model.zip" --lang C#

Note

Also may be useful to have AutoML logs packed inside (optionally) If model is trained and saved by Azure ML it may be useful to know how many minutes was spend on training and what models was tried...

zeahmed commented 5 years ago

@sergey-tihon, Thanks for bringing up this issue.

Currently, the trainer Code cannot be regenerated/retrieved from model.zip. This is inline with ML.NET model management philosophy. Trainer code and model.zip should be maintained together for future consumption.

It's not confirmed if there will be any such feature available in near future. I am currently tagging it as an enhancement for further discussion.

glebuk commented 5 years ago

The general issue is that the transitive closure of the training code is hard to capture in general, as a result, the proper way to deal with this is to integrate the traning into the CI pipeline, and store the code in the source code management system such as git repo and associate the model with the source code version hash.
For example, imagine that you have the actual training code recorded with the model, however, some 3rd-degree dependency such as ADO.NET got updated and now your code behaves slightly differently despite being unchanged.

najeeb-kazmi commented 5 years ago

Closing since this is a won't fix.