Histogram of scores produced by chex model

[lkourti]

Hello! We are using your chex model to evaluate the whole CheXpert-v1.0-small dataset. Here are the histograms of scores per disease: We were wondering if you could help us understand the jumps that happen on 0.5 for every disease. We read the relevant section in the paper, where you dicsuss the calibration of the model, but we don't fully understand how to connect this with the above question. Thank you in advance!

[ieee8023]

The outputs are calibrated: https://github.com/mlmed/torchxrayvision/blob/master/torchxrayvision/models.py#L219

It is discussed at the end of section 2 here: https://arxiv.org/abs/1901.11210 and section 3 here: https://arxiv.org/abs/2002.02497

For the chex model they were calibrated on a held out chex subset.

You can compute your own calibration thresholds with this script: https://github.com/mlmed/torchxrayvision/blob/master/scripts/model_calibrate.py

[ieee8023]

You can run the models uncalibrated if you set model.op_threshs = None as seen in this script: https://github.com/mlmed/covid-severity/blob/master/predict_severity.py

[lkourti]

That's helpful, thank you!