Could look like function(object, task, learner, ...), where
object is the output of predict(), should be of type Prediction
task is the original dataset, of type Dataset
learner is the model, of type Model ?
These parameters can of course be renamed
[ ] Implement printer generic for Evaluator class
print.Evaluator
Implement metrics
[ ] MAE
[ ] RMSE
[ ] AUC
[ ] Accuracy
[ ] Implement Evaluator for ResamplePrediction
Resample Prediciton is different since we have multiple prediction outputs
Maybe make Evaluator a generic, which has
Evaluator.Prediction
Evaluator.ResamplePrediction
Not sure how this would look like exaclty
Description
mlr.mini should provide a few functions that evaluate the performance of a prediction made by a model.
Commonly known measures are mean absolute error and root mean squared error for regression, and accuracy and area under the ROC curve (AUC) for classification.
Evaluators, like Inducers, should be functions with an S3 class for nicer printing.
Most evaluators only need the result made by predict(), but others could also need to access other information, like the original dataset, or the trained model.
They should therefore be functions of the form
function(object, task, learner, ...)
Like Inducers, Evaluators should be available through an evl environment:
mae <- evl$mae
identical(mae, EvaluatorMAE)
#> [1] TRUE
mae
#> Evaluator: Mean Absolute Error
#> Configuration: ()
mae(prediction)
#> [1] 6.5
Tasks
Evaluatorclass generatorInducerfunction(object, task, learner, ...), whereobjectis the output ofpredict(), should be of typePredictiontaskis the original dataset, of typeDatasetlearneris the model, of typeModel?Evaluatorclassprint.EvaluatorEvaluatorforResamplePredictionEvaluatora generic, which hasEvaluator.PredictionEvaluator.ResamplePredictionDescription
mlr.minishould provide a few functions that evaluate the performance of a prediction made by a model. Commonly known measures are mean absolute error and root mean squared error for regression, and accuracy and area under the ROC curve (AUC) for classification.Evaluators, like Inducers, should be functions with an
S3class for nicer printing. Most evaluators only need the result made bypredict(), but others could also need to access other information, like the original dataset, or the trained model. They should therefore be functions of the formLike Inducers, Evaluators should be available through an
evlenvironment: