[AutoML,CLI,ModelBuilder] Exploration Time vs Epochs + Loss/AUC (Enhancement)

[danucalovj]

This is more of a feature enhancement rather than an issue. Please let me know if this is the correct venue to submit this to.

In both the Model Builder GUI and the CLI for AutoML, the only option to customize training time is through the max exploration time setting. Example of CLI options:

This is a really neat way of changing things around instead of using epochs, using a time-based approach. But this could lead to undertraining a data set fairly easily, especially large ones, or produce a model that due to the fact that it hasn't been trained long enough (through several epochs throughout the entire dataset) didn't produce a good accuracy rate as expected.

I'm going to take a wild guess (can't find documentation for this) that the time based approach takes into account AUC or Loss as a criteria (or another common metric) to stop the training, is that correct? In other words if I have a small dataset and I set a training time of 4 hours, but the AUC or Loss didn't improve over X iterations it calls it a day and gives me the best result, is that correct?

It would be nice if the AutoML function (Model Builder, CLI and API) provided a way to (in addition to the time-based approach, or as an alternative to):

1. Set epochs for training rather than time based approach.
2. Set metric to look for when stopping a training, i.e.: auc, loss, etc.
3. Set thresholds of metrics to stop a training. i.e.: if after X number of epochs it doesn't improve, call it a day.

[danucalovj]

Here's an example to expand on this request... I set the following experiment to run for 1200 seconds, note the difference in results using the AveragePerceptronBinary trainer vs SgdCalibratedBinary, and their run times:

If I would have chosen to run this at half the time, my best result would have been the SgdCalibratedBinary trainer, with a 6 point loss in accuracy, 8 point in AUC. Also see iteration count column.

[CESARDELATORRE]

@danucalovj - thanks for great feedback. Sure, we’re planning on this kind of improvement, it is in our backlog. We’re thinking about dotnet/machinelearning#2 and dotnet/machinelearning#3:

dotnet/machinelearning#2 Set metric to look for when stopping a training, i.e.: auc, loss, etc.

dotnet/machinelearning#3 Set thresholds of metrics to stop a training. i.e.: if after X number of epochs it doesn't improve, call it a day.

When we have dotnet/machinelearning#3 implemented, then the by default—max-exploration-time will be very large and usually you’ll never get there, so it will exit if when additional iterations are not practically improving or if the user press a key combination to exit and we stop/generate the best model so far.

Or if the user specified a shorter explicit exploration time, of course.

[danucalovj]

Thanks @CESARDELATORRE

Since it is in your backlog, should I close this issue then?

[CESARDELATORRE]

@danucalovj - No, let’s keep it open until implemented. Open issues by users/customers are important :)