Week 8 Summary and Questions -- QSA Experimentation on the HPC

[dashaasienga]

@katcorr

Overview

This week I was mostly focused on thinking about some loose ends and how to tie them up as we move forward as well as filling some gaps in understanding before moving onto the next phase of the thesis work. It's also raised some questions on what the most important things to be thinking about are in the next coming weeks. It's dawning on me that the semester is almost coming to an end! Time is moving so fast!

In the next few weeks, I hope to fully cement my understanding of:

1. The problems studying fairness ML solves (can use COMPAS as a motivation).
2. Fairness definitions and methodologies both in a classification and regression framework.
3. The theory and implementation of the Seldonian algorithm.
4. The Seldonian algorithm in a regression setting.
5. Conflicts in fairness definitions (as I wrap up the regression setting and as a precursor to working with the Seldonian algorithm in the classification setting as the next step).

Big Picture Questions

1. The fall thesis talks will be happening in exactly a month! With the week in Thanksgiving lodged in there, I really only have about 3 full weeks to start synthesizing what I'd like to present. I'd love to exchange some ideas as well as what the format of that may look like.
2. Along those lines, I know that the physical deliverable of the thesis work this semester is Chapter 1. When would that be due by? When would be an ideal time to start seriously thinking about the format and content of that? Any advice on putting that together?
3. At the moment, it looks like we're moving in the direction of performing simulations to understand the fairness definition conflicts under different scenarios and (maybe) assessing long-term impact as well. Any preliminary thoughts on this?

Wrapping up Regression

Experimentation:

I was able to run all the experiments on the HPC and save the results! I'm having trouble, however, analyzing the results and creating the plots the way the researchers did :( I'm working on troubleshooting that and hoping to have that completed by our meeting time.

Application:

I found the GPA data set that the researchers used: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/O35FW8

It may be an interesting result to show the implementation of the Seldonian algorithm in a regression setting since the toy example we used is in a regression setting as well. As we noted, the toy-example was a bit impractical in the real world because we're constraining the MSE to fall within specific bounds. Since the thesis motivation is to use the Seldonian Algorithm to address fairness problems in machine learning, it may be worthwhile to have a practical example showing how this works in a fairness context before we move onto classification. We may be able to show some of the conflicts as well. We can look into this together and think more about that.

This link may also offer some useful resources on how to do this, particularly item D: https://seldonian.cs.umass.edu/Tutorials/tutorials/

Specific Questions

1. Thanks for your explanation on the formula on fairness conflicts! Though there are a lot of pieces to keep track of, it was very intuitive and easy to follow and it's definitely a very important result! This is something we can illustrate with the GPA data set and the COMPAS data set. I was wondering, how did you create the plots? They really helped to explain the concepts.

2. I'd like to re-visit this since we didn't get to it last time:

   

For context,

1. Independence: Decisions should be independent of any sensitive attribute. Also known as demographic parity, statistical parity, or group fairness. Focuses on equality of the predictions.
2. Separation: This is known as equality of odds (same type I and type II errors). Focuses on equality of the error rates.
3. Sufficiency: Takes the perspective of people that are given the same model decision and requires parity among them irrespective of sensitive features i.e. the precision should be the same across sensitive groups. Also known as predictive parity.

Note that S here refers to the probability score/ prediction (i.e. the probability that the observations fall in the positive class).

There's a lot more we can discuss regarding these 3 categories of fairness definitions. Key literature source: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8913820/

Key Next Steps

1. Complete experimentation of the QSA with synthetic regression data
2. Implement a QSA algorithm to predict the GPAs of students, with gender and/ or race as the protected attribute [I'll want to compare these results with OLS]
3. Start thinking ahead about the fall thesis talks and Chapter 1 of the thesis

Once this is wrapped up, we can move on to the classification setting where I envision we'll run the main simulations and main application with the COMPAS data set.

[katcorr]

Yes! I was just thinking today how close we are to end of the semester and we should talk about the Ch 1 expectations. We can talk about all this tomorrow when we meet, but I did just want to mention that I uploaded this R script which contains the code I used to create the PPV, NPV, FPR plots. Did it quickly so no comments and disorganized, but I can talk through tomorrow and update it . . .

[dashaasienga]

Yes, let's definitely talk through the R Script and expectations for Chapter 1 + Fall presentations tomorrow!

[dashaasienga]

Update: I was able to reproduce the experimentation, though I still need to clean up/ organize the notebook and understand all the code. Quick question, I don't know how to push the files from the Jupyter HPC to Github and I don't see an option to download to my local computer first, and then push :( I'm wondering if you happen to know how to, and I may also be missing something. We can ask Andy about this as well.