Metric Testing + Implementation - Githubissues

ChEB-AI / python-chebai

GNU Affero General Public License v3.0

12 stars 4 forks source link

Metric Testing + Implementation #27

Closed aditya0by0 closed 6 months ago

aditya0by0 commented 6 months ago

Pull Request for Issue number #22

Test the Macro-F1

[x] Compare the result for iteratively updating (i.e., passing batches to the update function) and calculation via a single call
[x] Compare the custom implementation to the torchmetrics MultilabelF1Score
[x] Cover different datasets
- trivial cases: e.g. all predictions are 1 and half the labels are 1, all the labels are 1 and half the predictions are 1
- case where some classes have no labels
- realistic case with data from model output

Add balanced accuracy metric

[x] Add implementation of the balanced accuracy ba = (tpr + tnr)/2 = (tp/(tp+fn) + tn(tn+fp))/2
[x] Add tests (similar to macro-f1)