bglocker
commented
10 months ago Many thanks for the kind feedback!
Note, when we conduct the bias analysis with equal samples (n=3000) from each group, this is after resampling the test data where we control for age and disease prevalance across groups (see chexpert.resample.ipynb).
I would share your concerns regarding age bias. It is an interesting but possibly more difficult characteristic to analyse as it is highly correlated with disease. Not sure what the best pre-processing/resampling would be in that case, as there may be other confounders, as well.
More generally, we believe that bias analysis across subgroups that have different base statistics in the original data is tricky, as we often do not know where these differences come from and we need to make strong assumptions when applying test set resampling.
Dear Authors,
Great work on the paper, and thanks for providing the code!
I have a question regarding the sampling technique when performing bias analysis. I've noticed that you took 3000 samples containing equal fractions of "white", "asian" and "black" subgroups.
My first question is, how do you ensure that you have equal fractions of healthy/disease? The notebooks provided don't seem to take that into account.
My second question is, when evaluating the bias for age, for example, do we need to prepare a different sample with equal number of age groups (<30, 30-40, etc.)? My concern is that certain age groups will be underrepresented and performing bias analysis on them will not be correctly reflected.
Hope its clear, thanks in advance!