Advanced Training Options - Optimization Metrics / Exclude algorithms

[luisquintanilla]

In the Model Buidler training tab, provide a dialog for "Advanced Training Options", similar to the existing "Advaced Data Options" in the data tab. In this dialog, a user should be able to:

• Select which metric they want to optimize for related to the scenario they've chosen (i.e. Value Prediction = R-Squared, Data classification = Micro-accuracy)
• Choose which algorithms to exclude from training

[luisquintanilla]

@zewditu - UI @LittleLittleCloud - Backend

[luisquintanilla]

This issue is blocking #60 @luisquintanilla to provide design by EOW.

[luisquintanilla]

To-Do's

@luisquintanilla

• [ ] Provide copy for each of the scenarios in the Trainers Tab.

@LittleLittleCloud

• [x] Provide list of trainers used for Data Classification and Value Prediction scenarios. (provided below)
• [x] Confirm image classification uses the same metrics as (multiclass) classification.
• [x] Confirm forecasting metrics used are the same as regression (value prediction). Does forecasting use R-Squared as one of the optimization metrics? forecasting using a set of different metrics, I've updaed below.

@zewditu

• [ ] Confirm whether Azure ML supported scenarios (image classification / object detection) allow for setting the evaluation metric.

Proposed Design

Copy

Optimizing Metric Tab

Note: For each of the scenarios, set the default to the metric being used today.

Heading: Choose the metric that will be used to find the best-performing model. Subheading: When should I use each metric?

Data classification

Binary Classification

Note: This is determined based on whether the label column is boolean.

• Accuracy - The closer to 1.00, the better.
  • The ratio of number of correct predictions to the total number of input samples.
• Area under the curve (AUC) - The closer to 1.00, the better.
  • The area under the curve created by sweeping the true positive rate vs. the false positive rate.
• Area under the curve of a Precision-Recall curve (AUCPR) - The closer to 1.00, the better.
  • Useful measure of success of prediction when the classes are imbalanced (highly skewed datasets).
• F1-score - The closer to 1.00, the better.
  • It's the harmonic mean of the precision and recall.

Multiclass Classification

• Micro-Accuracy - The closer to 1.00, the better.
  • Aggregates the contributions of all classes to compute the average metric. Doesn't take into account class membership.
• Macro-Accuracy - The closer to 1.00, the better.
  • The average accuracy at the class level. Takes into account class membership.
• Logarithmic loss (Log-Loss) - The closer to 0.00, the better.
  • The performance of a classification model where the prediction input is a probability value between 0.00 and 1.00.
• Logarithmic loss reduction (Log-Loss Reduction) - Ranges from -inf and 1.00, where 1.00 is perfect predictions and 0.00 indicates mean predictions.
  • The advantage of the classifier over a random prediction.

Value prediction

• R-Squared - The closer to 1.00, the better quality.
  • How close the actual data values are to the predicted values.
• Mean Absolute Error (MAE) - The closer to 0.00, the better quality.
  • The average of all the model errors, where model error is the absolute distance between the predicted value and the actual value.
• Mean Squared Error (MSE) - It is always non-negative, and values closer to 0.00 are better.
  • The squared average of all model errors, where model error is the absolute distance between the predicted value and the actual value.
• Root Mean Squared Error (RMSE) - It is always non-negative, and values closer to 0.00 are better.
  • The squared root of the mean squared error of all model errors, where model error is the absolute distance between the predicted value and the actual value.

Image classification

Same as data classification (multiclass). Disable for cloud scenario. Azure ML environment does not allow to specify metric.

Recommendation

Same as value prediction.

Object Detection

Disabled for now. Azure ML scenario does not allow to specify metric.

Forecasting

• RMSE
• MAPE ( >= 0, smaller the better)
  • calculates the average error over time
• MAE ( >= 0, the smaller the better)

Trainers Tab

Heading: Choose the trainers to explore during training. Subheading: Which trainer should I use?

Data classification

• Stochastic dual coordinated ascent (SDCA)
  • Works well for features that are linearly separable. Tuning not needed for good default performance.
• L-BFGS
  • Works well for features that are linearly separable. Use when number of features is large.
• Light gradient boosted machine (LightGbm)
  • Features do not need to be linearly separable to use this type of algorithm. Fastest and most accurate of tree trainers. Highly tunable.
• Fast Tree
  • Features do not need to be linearly separable to use this type of algorithm. Works well with noisy data.
• Fast Forest
  • Features do not need to be linearly separable to use this type of algorithm. Resilient to unbalanced data. Highly tunable.

Value prediction

• Stochastic dual coordinated ascent (SDCA)

  • Works well for features that are linearly separable. Tuning not needed for good default performance.

• L-BFGS

  • Works well for features that are linearly separable. Use when number of features is large.

• Light gradient boosted machine (LightGbm)

  • Features do not need to be linearly separable to use this type of algorithm. Fastest and most accurate of tree trainers. Highly tunable Features do not need to be linearly separable to use this type of algorithm. Boosting algorithm based on decision tree. Highly tunable

• Fast Tree

  • Features do not need to be linearly separable to use this type of algorithm. Works well with noisy data. Highly tunable.

• Fast Forest

  • Features do not need to be linearly separable to use this type of algorithm. Resilient to unbalanced data. Highly tunable.

Image classification

• ImageClassification API Disabled for now: ImageClassification and Azure ML are the only options depending on the environment (Local vs. Cloud).

Recommendation

Disabled for now. MatrixFactorization is the only option

Object Detection

Disabled for now. Azure ML is the only option.

Forecasting

Disabled for now. ForecastBySsa is the only option.

[zewditu]

For Image classification : Accuracy, Average Precision Score Weight, AUC weighted , normalized macro recall For OD: we only have one, MeanAveragePrecision.

[zewditu]

One thing want to confirm, when we say " Image classification metrices are the same as data classification", are we talking about the multi-classification one?

[luisquintanilla]

One thing want to confirm one thing , when we say " Image classification metrices are the same as data classification", are we talking about the muti-classification one?

Good question. It looks like it's multiclass. Here's a model I just trained. It uses MicroAccuracy.

|      Trainer                             MicroAccuracy Duration    |
|--------------------------------------------------------------------|
|0     ImageClassificationMulti            0.8564     520.1900       |
|--------------------------------------------------------------------|
|                          Experiment Results                        |
|--------------------------------------------------------------------|
|                               Summary                              |
|--------------------------------------------------------------------|
|ML Task: image classification                                       |
|Dataset: C:\Users\luquinta.REDMOND\Datasets\EuroSAT200              |
|Label : Label                                                       |
|Total experiment time :   520.1900 Secs                             |
|Total number of models explored: 1                                  |
|--------------------------------------------------------------------|
|                        Top 1 models explored                       |
|--------------------------------------------------------------------|
|      Trainer                             MicroAccuracy Duration    |
|--------------------------------------------------------------------|
|0     ImageClassificationMulti            0.8564     520.1900       |
|--------------------------------------------------------------------|

[luisquintanilla]

@LittleLittleCloud can you please take a look at the descriptions I provided for the trainers to make sure I accurately captured the main points.

@luisquintanilla overall looks good, made one small change on lightGbm though