search

metriculous-ml / metriculous

Measure and visualize machine learning model performance without the usual boilerplate.
MIT License
94 stars 11 forks source link

Manually specify labels for cm #3

Closed magnusja closed 3 years ago

magnusja commented 3 years ago

When y_true contains [ 0 1 2 3 5 6 7 8 9 11 12] and y_pred [0 1 3 5 6 8 9] ie. my model does not predict any of the classes 2, 4, ...

I get this error:

~/Library/Caches/pypoetry/virtualenvs/document-classification-ReJIVt9a-py3.8/lib/python3.8/site-packages/metriculous/__init__.py in compare_classifiers(ground_truth, model_predictions, model_names, sample_weights, class_names, one_vs_all_quantities, one_vs_all_figures, top_n_accuracies, filter_quantities, filter_figures, primary_metric, simulated_class_distribution, class_label_rotation_x, class_label_rotation_y)
     48 ) -> Comparison:
     49 
---> 50     return compare(
     51         evaluator=ClassificationEvaluator(
     52             class_names=class_names,

~/Library/Caches/pypoetry/virtualenvs/document-classification-ReJIVt9a-py3.8/lib/python3.8/site-packages/metriculous/_comparison.py in compare(evaluator, ground_truth, model_predictions, model_names, sample_weights)
    169         assert_that(model_names).is_length(len(model_predictions))
    170 
--> 171     model_evaluations = [
    172         evaluator.evaluate(
    173             ground_truth,

~/Library/Caches/pypoetry/virtualenvs/document-classification-ReJIVt9a-py3.8/lib/python3.8/site-packages/metriculous/_comparison.py in <listcomp>(.0)
    170 
    171     model_evaluations = [
--> 172         evaluator.evaluate(
    173             ground_truth,
    174             model_prediction=pred,

~/Library/Caches/pypoetry/virtualenvs/document-classification-ReJIVt9a-py3.8/lib/python3.8/site-packages/metriculous/evaluators/_classification_evaluator.py in evaluate(self, ground_truth, model_prediction, model_name, sample_weights)
    177 
    178         # === Figures ==================================================================
--> 179         unfiltered_lazy_figures = self._lazy_figures(
    180             model_name,
    181             data=data,

~/Library/Caches/pypoetry/virtualenvs/document-classification-ReJIVt9a-py3.8/lib/python3.8/site-packages/metriculous/evaluators/_classification_evaluator.py in _lazy_figures(self, model_name, data, maybe_sample_weights, class_names)
    276                 (
    277                     "Confusion Matrix",
--> 278                     _bokeh_confusion_matrix(
    279                         y_true=y_true,
    280                         y_pred=y_pred,

~/Library/Caches/pypoetry/virtualenvs/document-classification-ReJIVt9a-py3.8/lib/python3.8/site-packages/metriculous/evaluators/_classification_figures_bokeh.py in _bokeh_confusion_matrix(y_true, y_pred, class_names, title_rows, x_label_rotation, y_label_rotation)
    177             predicted.append(j_class)
    178             actual.append(i_class)
--> 179             count.append(cm[i, j])
    180             normalized.append(cm_normalized[i, j])
    181             normalized_by_pred.append(cm_normalized_by_pred[i, j])

IndexError: index 11 is out of bounds for axis 1 with size 11
marlonjan commented 3 years ago

Perfect. CI breaks, but that's not your fault, I will take a look and get this merged. Thanks 👌

magnusja commented 3 years ago

Added NaN to zero conversion otherwise I get a bokeh exception complaining about out of range floats unsupported in json