Matchzoo metrics only works on ranking

[yunhenk]

Describe the Question

this happens when do evaluation

  File "/usr/local/lib/python3.6/dist-packages/MatchZoo-2.1.0-py3.6.egg/matchzoo/engine/base_model.py", line 333, in evaluate
    raise ValueError("Matchzoo metrics only works on ranking.")

and here is my task, following the comment of Classification class

    classify_task = mz.tasks.Classification()
    classify_task.metrics = ['precision']

please help check this problem,thanks

[uduse]

So what is your question? Why matchzoo metrics do not work for classification?

[yunhenk]

So what is your question? Why matchzoo metrics do not work for classification?

Yes, what metric should i use for classification? Do i need to write one myself?

[yunhenk]

Hi, I checked some source code, here is : BaseModel.evaluate:

    def evaluate(
        self,
        x: typing.Dict[str, np.ndarray],
        y: np.ndarray,
        batch_size: int = 128
    ) -> typing.Dict[BaseMetric, float]:
        result = dict()
        matchzoo_metrics, keras_metrics = self._separate_metrics()
        y_pred = self.predict(x, batch_size)

        for metric in keras_metrics:
            metric_func = keras.metrics.get(metric)
            result[metric] = K.eval(K.mean(
                metric_func(K.variable(y), K.variable(y_pred))))

        if matchzoo_metrics:
            if not isinstance(self.params['task'], tasks.Ranking):
                raise ValueError("Matchzoo metrics only works on ranking.")
            for metric in matchzoo_metrics:
                result[metric] = self._eval_metric_on_data_frame(
                    metric, x['id_left'], y, y_pred)

        return result

As far as i can see, the precision metric is a successor of BaseMetric, ie. a matchzoo_metric, which makes it unable to be used on classification tasks, this seems unreasonable.

[bwanglzu]

@yunhenk in this case you should make use of keras metrics, see here

Just noticed that precision and recall has been removed from keras metrics, you can custom your metrics like this:

def precision(y_true, y_pred):
    """Precision metric.
    Only computes a batch-wise average of precision.
    Computes the precision, a metric for multi-label classification of
    how many selected items are relevant.
    """
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision

def recall(y_true, y_pred):
    """Recall metric.
    Only computes a batch-wise average of recall.
    Computes the recall, a metric for multi-label classification of
    how many relevant items are selected.
    """
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall

model.compile(optimizer='sgd',
              loss='binary_crossentropy',
              metrics=[precision, recall])

[yunhenk]

@bwanglzu Thanks, got it. By the way, do you consider adding metrics for classification tasks, since keras metrics removed now. And add one classification task tutorial maybe?

[bwanglzu]

@yunhenk We'll add tutorials for classification, need some discussion whether to add precision/recall into metrics, might confused with ranking metrics..

hopefully your questions has been resolved, I'll close this issue for now, if you have future questions please feel free to open another one.

[Sjzwind]

@yunhenk We'll add tutorials for classification, need some discussion whether to add precision/recall into metrics, might confused with ranking metrics..

hopefully your questions has been resolved, I'll close this issue for now, if you have future questions please feel free to open another one.

there is no any tutorials about classification until today， three months has gone