Metrics implementations need to be tested / documented - Githubissues

ChEB-AI / python-chebai

GNU Affero General Public License v3.0

11 stars 4 forks source link

Metrics implementations need to be tested / documented #22

Closed sfluegel05 closed 3 months ago

sfluegel05 commented 4 months ago

Currently, we use the torchmetrics library for metrics. However, we also have a custom implementation for the macro-f1.

Test the Macro-F1

[x] Compare the result for iteratively updating (i.e., passing batches to the update function) and calculation via a single call
[x] Compare the custom implementation to the torchmetrics MultilabelF1Score
[x] Cover different datasets
- trivial cases: e.g. all predictions are 1 and half the labels are 1, all the labels are 1 and half the predictions are 1
- case where some classes have no labels
- realistic case with data from model output

Document implementation

[ ] Add wiki page that covers the following questions:
- Which metrics are used in ChEB-AI by default?
- How do I change which metrics get tracked during training / validation?
- How do I change which metrics appear in the names of checkpoints?
- How do I add my own metric (either from torchmetrics or with custom implementation)?

Add balanced accuracy metric

[x] Add implementation of the balanced accuracy ba = (tpr + tnr)/2 = (tp/(tp+fn) + tn(tn+fp))/2
[x] Add tests (similar to macro-f1)