search

netrack / keras-metrics

Metrics for Keras. DEPRECATED since Keras 2.3.0
MIT License
164 stars 23 forks source link

Feature request: Multi-class average recall metric #33

Closed YiqinZhao closed 5 years ago

YiqinZhao commented 5 years ago

Hi! I have been using multi-class average recall in most of my research projects in the past. I used to calculate them by callbacks, now I can finally add a metric.

I have posted my implementation at this gist. Would be convenient to have a look at my work? If you are interested in it, I will be glad to submit a PR.

The multi-class average recall is a widely used metric in emotion classification and many other research problems. I think it could certainly benefit lots of people.

ybubnov commented 5 years ago

Hi @HawkinsZhao, I will definitely appreciate to see this feature in the keras-metrics. My only wish is unit tests for that metric.

YiqinZhao commented 5 years ago

Hi @ybubnov. I would like to create a unit test file. But before that, I wanna discuss a bit more. I have noticed the current metrics are designed for 2-classes problem, but with a little modification on my code, we could easily support multi-classes problem. We could use my code to calculate the average precision/recall or single label precision/recall. Also, I have noticed some issues discussing this, eg #22.

So, I would like to know, what is your plan for supporting multi-classes? Do you think we should redesign the unit tests for better multi-classes support?

By the way, the Keras team is redesigning the API for metrics recently, it will be included in the future release. After it is stable, do you think we should merge this repo into the Keras core repo?

ybubnov commented 5 years ago

I have noticed the current metrics are designed for 2-classes problem ...

Current implementation allows to specify a label (read class) which should be evalueated by the metric. For example, a metric that captures precision for the output class 3 looks like this:

km.categorical_precision(label=3)

But you're right, this library is missing support of averaging the metric over the multiple classes.

So, I would like to know, what is your plan for supporting multi-classes? Do you think we should redesign the unit tests for better multi-classes support?

I'm not against of any improvements.

By the way, the Keras team is redesigning the API for metrics recently, it will be included in the future release. After it is stable, do you think we should merge this repo into the Keras core repo?

The main reason why I'm not trying to include these metrics into the main keras package is that they have a huge code-base and adding a new metric or changing existing implementation leads to bureaucracy that is common for big projects. To be straight: we can release keras-metrics more frequently that keras.

But again, I'm not insisting on keeping development of this library separate from the keras. Probably, it would be better to talk to keras-team maintainers about this first.

YiqinZhao commented 5 years ago

You are right. Let's stay with keras-metrics for now. I am closing this issue, and I will submit a PR later.