How to count accuracy in Chestxray14 dataset?

[wenxin06133]

Please teach us how to add code to count the accuracy ,precision, recall and also confision matrix of mutilple labels!! Thanks alot!!

[Mda233]

Thanks for your attention to our work.

There is an test_classification() method in the trainer.py which can return the y_true and y_pred for all the testing samples. Then you can use the values from these two tensors (dimension: num of samples * 14 classes) for your calculation. You can refer to the final part of the engine.py to see how we calculate AUC. For accuracy ,precision, recall and confision matrix, it would be similar as AUC by using sklearn.metrics methods.

Hope this helps.

[wenxin06133]

I used your pre-trained weights and train on my own data. I found that mean AUC was high(75%),but the tensor of y_pred only tend to one class. Whether the problem is because the model was originally designed for a single class instead of multiple classes? Do you have any suggestions?

Mda233 @.***> 於 2022年11月1日 週二 上午9:47寫道：

Thanks for your attention to our work.

There is an test_classification() method in the trainer.py which can return the y_true and y_pred for all the testing samples. Then you can use the values from these two tensors (dimension: num of samples * 14 classes) for your calculation. You can refer to the final part of the engine.py to see how we calculate AUC. For accuracy ,precision, recall and confision matrix, it would be similar as AUC by using sklearn.metrics methods.

Hope this helps.

— Reply to this email directly, view it on GitHub https://github.com/jlianglab/BenchmarkTransformers/issues/2#issuecomment-1297906409, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2FS2IW53CVNU5EHPRDFCJTWGBZDVANCNFSM6AAAAAARTTSM5E . You are receiving this because you authored the thread.Message ID: @.***>

[Mda233]

You may want to check your dataset. There might be an issue of data imbalance of positive (labeled as 1) and negetive (labeled as 0) samples. This is a common issue for medical datasets. The model tends to predict small values to make the loss smaller.

[wenxin06133]

sorry for asking question again!! I want to ask how you seperated chestxray14 dataset into three parts? Because it is muiltilabels ,it might be an issue of data imbalance.

[Mda233]

We use the official data split of chestxray14 dataset for training, validating and testing.