Note: the current releases of this toolbox are a beta release, to test working with Haskell's, Python's, and R's code repositories.
Metrics provides implementations of various supervised machine learning evaluation metrics in the following languages:
- Python
easy_install ml_metrics - R
install.packages("Metrics")from the R prompt - Haskell
cabal install Metrics - MATLAB / Octave (clone the repo & run setup from the MATLAB command line)
For more detailed installation instructions, see the README for each implementation.
EVALUATION METRICS
| Evaluation Metric | Python | R | Haskell | MATLAB / Octave |
| Absolute Error (AE) | ✓ | ✓ | ✓ | ✓ |
| Average Precision at K (APK, AP@K) | ✓ | ✓ | ✓ | ✓ |
| Area Under the ROC (AUC) | ✓ | ✓ | ✓ | ✓ |
| Classification Error (CE) | ✓ | ✓ | ✓ | ✓ |
| F1 Score (F1) | ✓ | |||
| Gini | ✓ | |||
| Levenshtein | ✓ | ✓ | ✓ | |
| Log Loss (LL) | ✓ | ✓ | ✓ | ✓ |
| Mean Log Loss (LogLoss) | ✓ | ✓ | ✓ | ✓ |
| Mean Absolute Error (MAE) | ✓ | ✓ | ✓ | ✓ |
| Mean Average Precision at K (MAPK, MAP@K) | ✓ | ✓ | ✓ | ✓ |
| Mean Quadratic Weighted Kappa | ✓ | ✓ | ✓ | |
| Mean Squared Error (MSE) | ✓ | ✓ | ✓ | ✓ |
| Mean Squared Log Error (MSLE) | ✓ | ✓ | ✓ | ✓ |
| Normalized Gini | ✓ | |||
| Quadratic Weighted Kappa | ✓ | ✓ | ✓ | |
| Relative Absolute Error (RAE) | ✓ | |||
| Root Mean Squared Error (RMSE) | ✓ | ✓ | ✓ | ✓ |
| Relative Squared Error (RSE) | ✓ | |||
| Root Relative Squared Error (RRSE) | ✓ | |||
| Root Mean Squared Log Error (RMSLE) | ✓ | ✓ | ✓ | ✓ |
| Squared Error (SE) | ✓ | ✓ | ✓ | ✓ |
| Squared Log Error (SLE) | ✓ | ✓ | ✓ | ✓ |
TO IMPLEMENT
- F1 score
- Multiclass log loss
- Lift
- Average Precision for binary classification
- precision / recall break-even point
- cross-entropy
- True Pos / False Pos / True Neg / False Neg rates
- precision / recall / sensitivity / specificity
- mutual information
HIGHER LEVEL TRANSFORMATIONS TO HANDLE
- GroupBy / Reduce
- Weight individual samples or groups
PROPERTIES METRICS CAN HAVE
(Nonexhaustive and to be added in the future)
- Min or Max (optimize through minimization or maximization)
- Binary Classification
- Scores predicted class labels
- Scores predicted ranking (most likely to least likely for being in one class)
- Scores predicted probabilities
- Multiclass Classification
- Scores predicted class labels
- Scores predicted probabilities
- Regression
- Discrete Rater Comparison (confusion matrix)