Closed yunhenk closed 5 years ago
So what is your question? Why matchzoo metrics do not work for classification?
So what is your question? Why matchzoo metrics do not work for classification?
Yes, what metric should i use for classification? Do i need to write one myself?
Hi, I checked some source code, here is : BaseModel.evaluate:
def evaluate(
self,
x: typing.Dict[str, np.ndarray],
y: np.ndarray,
batch_size: int = 128
) -> typing.Dict[BaseMetric, float]:
result = dict()
matchzoo_metrics, keras_metrics = self._separate_metrics()
y_pred = self.predict(x, batch_size)
for metric in keras_metrics:
metric_func = keras.metrics.get(metric)
result[metric] = K.eval(K.mean(
metric_func(K.variable(y), K.variable(y_pred))))
if matchzoo_metrics:
if not isinstance(self.params['task'], tasks.Ranking):
raise ValueError("Matchzoo metrics only works on ranking.")
for metric in matchzoo_metrics:
result[metric] = self._eval_metric_on_data_frame(
metric, x['id_left'], y, y_pred)
return result
As far as i can see, the precision metric is a successor of BaseMetric, ie. a matchzoo_metric, which makes it unable to be used on classification tasks, this seems unreasonable.
@yunhenk in this case you should make use of keras metrics, see here
Just noticed that precision and recall has been removed from keras metrics, you can custom your metrics like this:
def precision(y_true, y_pred):
"""Precision metric.
Only computes a batch-wise average of precision.
Computes the precision, a metric for multi-label classification of
how many selected items are relevant.
"""
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
precision = true_positives / (predicted_positives + K.epsilon())
return precision
def recall(y_true, y_pred):
"""Recall metric.
Only computes a batch-wise average of recall.
Computes the recall, a metric for multi-label classification of
how many relevant items are selected.
"""
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
recall = true_positives / (possible_positives + K.epsilon())
return recall
model.compile(optimizer='sgd',
loss='binary_crossentropy',
metrics=[precision, recall])
@bwanglzu Thanks, got it. By the way, do you consider adding metrics for classification tasks, since keras metrics removed now. And add one classification task tutorial maybe?
@yunhenk We'll add tutorials for classification, need some discussion whether to add precision/recall into metrics, might confused with ranking metrics..
hopefully your questions has been resolved, I'll close this issue for now, if you have future questions please feel free to open another one.
@yunhenk We'll add tutorials for classification, need some discussion whether to add precision/recall into metrics, might confused with ranking metrics..
hopefully your questions has been resolved, I'll close this issue for now, if you have future questions please feel free to open another one.
there is no any tutorials about classification until today, three months has gone
Describe the Question
this happens when do evaluation
and here is my task, following the comment of Classification class
please help check this problem,thanks