search

cran / wconf

:exclamation: This is a read-only mirror of the CRAN R package repository. wconf — Weighted Confusion Matrix. Homepage: https://www.alexandrumonahov.eu.org/projects
0 stars 0 forks source link

"balancedaccuracy" does not work well with weighted confusion matrix #1

Open ElenaBrc opened 1 month ago

ElenaBrc commented 1 month ago

Hello Alexandru,

I found your paper about wconf very clear and interesting. Nevertheless I found an issue I was hoping you could help me with. In the description of your package you state the following: "Notice, however, that both the SinACC and BalACC scores are invariant to the distance of the predicted value from the correct category. If there is value in assigning some positive weight to predictions classified in the vicinity of the correct category or, conversely, applying a supplementary penalty to predictions situated far away from the correct category, then you should consider first applying weights to the confusion matrix using the function “wconfusionmatrix”, and then using the “balancedaccuracy” function on the weighted matrix"

I was having that same problem and I tried to do that. The problem is that the balancedaccuracy function does not consider that you are providing a weighted matrix. Let me show you with an example, with custom weights:

cm<-wconfusionmatrix(x[["table"]],
  • weight.type = "custom", custom.weights =c(1,0,0,0,0),
  • weight.penalty = FALSE,
  • print.weighted.accuracy = TRUE) Weighted accuracy = 0.3850702

when I appy the balancedaccuracy function I obtain this: Overall accuracy metrics: SinACC = 1 BalACC = 1 ACC = 1

As you can see, this function is considering the weights as total values, therefore giving accuracy metrics that have nothing to do with the weighted confusion matrix.

I hope you can help me with this issue, Thanks in advance

gaborcsardi commented 1 month ago

Hi, this is a read only mirror of CRAN, please see the package authors in the DESCRIPTION file. Look for Maintainer, BugReports and URL. Thanks!