We think that True positivity parity (TPP) is a useful rate to tell us how correct the algorithm is compared to the ground truth. The only problem with this is that it is difficult to find out the FNs. Will require a training set based on real policy document data. This will need labelled data and a careful thought out test to be as valid as possible.
We also realise that although TPP would be an appropriate metric, we should check for parity between other performance metrics (list TBD) as there may be discrepancies that will indicate unfairness.
We think that True positivity parity (TPP) is a useful rate to tell us how correct the algorithm is compared to the ground truth. The only problem with this is that it is difficult to find out the FNs. Will require a training set based on real policy document data. This will need labelled data and a careful thought out test to be as valid as possible.
We also realise that although TPP would be an appropriate metric, we should check for parity between other performance metrics (list TBD) as there may be discrepancies that will indicate unfairness.
to-do: Liz to think of appropriate test method