The Epic Road to Fairness

[Hamedloghmani]

The plan for both short and long term future development of this project will be discussed here. This is just a road map to have a bird's-eye view on the project and each task should be defined in details in another issue page if needed. In case a separate issue page is created for a task, it should be linked here by the number of created issue.

1- Experiments

• [x] Equality of Opportunity.popularity.0.01.100.fa-ir.uspt
• [x] Demographic Parity.popularity.0.01.100.fa-ir.uspt
• [x] Equality of Opportunity.gender.0.01.100.fa-ir.uspt
• [x] Demographic Parity.gender.0.01.100.fa-ir.uspt
• [x] Equality of Opportunity.popularity.0.05.100.fa-ir.uspt
• [x] Demographic Parity.popularity.0.05.100.fa-ir.uspt
• [x] Equality of Opportunity.gender.0.05.100.fa-ir.uspt
• [x] Demographic Parity.gender.0.05.100.fa-ir.uspt #######################

2- Implementation:

• [ ] Optimize fair_greedy
• [x] Add argument in get_stats to choose between avg or auc for pupolarity
• [x] Add det_const_sort to the pipeline

3- Refactor

• [ ] Edit Adila's Readme and add link to new full experiments ( for undergrad mentees)
• [ ] 2d scatter diagram for fairness vs utility #67
• [ ] Change popularity label files from csv to binary
• [ ] f'{new_output}.{algorithm}.{k_max}.rerank.csv' => binary
• [ ] Separate the layers of the pipeline (dal, rerank, eval layers)
• [ ] removing torch from pipeline and using pickle instead. Now, we add torch only to load *.pred file of a team formation model #64

4- Write up and literature review

Done:

• [x] Equality of Opportunity.popularity.0.1.100.fa-ir.uspt
• [x] Demographic Parity.popularity.0.1.100.fa-ir.uspt
• [x] Equality of Opportunity.gender.0.1.100.fa-ir.uspt
• [x] Demographic Parity.gender.0.1.100.fa-ir.uspt
• [x] Equality of Opportunity.popularity.100.det.uspt
• [x] Demographic Parity.popularity.100.det.uspt
• [x] Equality of Opportunity.gender.100.det.uspt
• [x] Demographic Parity.gender.100.det.uspt
• [x] Run all the gender related experiments on dblp after fixing the bug in dblp gender-to-index file

#######################

• [x] Equality of Opportunity.gender.det.imdb
• [x] Equality of Opportunity.gender.det.dblp
• [x] Demographic Parity.popularity.det.imdb
• [x] Equality of Opportunity.popularity.det.imdb ###################################################################
• [x] Equality of Opportunity.popularity.0.1.100.fa-ir.ndkl.imdb
• [x] Equality of Opportunity.popularity.0.1.100.fa-ir.skew.imdb
• [x] Equality of Opportunity.popularity.0.1.100.fa-ir.ndkl.dblp
• [x] Equality of Opportunity.popularity.0.1.100.fa-ir.skew.dblp
• [x] Equality of Opportunity.gender.0.1.100.fa-ir.ndkl.imdb
• [x] Equality of Opportunity.gender.0.1.100.fa-ir.skew.imdb
• [x] Equality of Opportunity.gender.0.1.100.fa-ir.ndkl.dblp
• [x] Equality of Opportunity.gender.0.1.100.fa-ir.skew.dblp
• [x] Demographic Parity.popularity.0.1.100.fa-ir.ndkl.imdb
• [x] Demographic Parity.popularity.0.1.100.fa-ir.skew.imdb
• [x] Demographic Parity.popularity.0.1.100.fa-ir.ndkl.dblp
• [x] Demographic Parity.popularity.0.1.100.fa-ir.skew.dblp
• [x] Demographic Parity.gender.0.1.100.fa-ir.ndkl.imdb
• [x] Demographic Parity.gender.0.1.100.fa-ir.skew.imdb
• [x] Demographic Parity.gender.0.1.100.fa-ir.ndkl.dblp
• [x] Demographic Parity.gender.0.1.100.fa-ir.skew.dblp

###################################################################

• [x] Equality of Opportunity.popularity.0.05.100.fa-ir.ndkl.imdb
• [x] Equality of Opportunity.popularity.0.05.100.fa-ir.skew.imdb
• [x] Equality of Opportunity.popularity.0.05.100.fa-ir.ndkl.dblp
• [x] Equality of Opportunity.popularity.0.05.100.fa-ir.skew.dblp
• [x] Equality of Opportunity.gender.0.05.100.fa-ir.ndkl.imdb
• [x] Equality of Opportunity.gender.0.05.100.fa-ir.skew.imdb
• [x] Equality of Opportunity.gender.0.05.100.fa-ir.ndkl.dblp
• [x] Equality of Opportunity.gender.0.05.100.fa-ir.skew.dblp
• [x] Equality of Opportunity.popularity.0.01.100.fa-ir.ndkl.imdb
• [x] Equality of Opportunity.popularity.0.01.100.fa-ir.skew.imdb
• [x] Equality of Opportunity.popularity.0.01.100.fa-ir.ndkl.dblp
• [x] Equality of Opportunity.popularity.0.01.100.fa-ir.skew.dblp
• [x] Equality of Opportunity.gender.0.01.100.fa-ir.ndkl.imdb
• [x] Equality of Opportunity.gender.0.01.100.fa-ir.skew.imdb
• [x] Equality of Opportunity.gender.0.01.100.fa-ir.ndkl.dblp
• [x] Equality of Opportunity.gender.0.01.100.fa-ir.skew.dblp

###################################################################

• [x] Demographic Parity.popularity.0.05.100.fa-ir.ndkl.imdb

• [x] Demographic Parity.popularity.0.05.100.fa-ir.skew.imdb

• [x] Demographic Parity.popularity.0.05.100.fa-ir.ndkl.dblp

• [x] Demographic Parity.popularity.0.05.100.fa-ir.skew.dblp

• [x] Demographic Parity.gender.0.05.100.fa-ir.ndkl.imdb

• [x] Demographic Parity.gender.0.05.100.fa-ir.skew.imdb

• [x] Demographic Parity.gender.0.05.100.fa-ir.ndkl.dblp (compute canada)

• [x] Demographic Parity.gender.0.05.100.fa-ir.skew.dblp (compute canada)

• [x] Demographic Parity.popularity.0.01.100.fa-ir.ndkl.imdb

• [x] Demographic Parity.popularity.0.01.100.fa-ir.skew.imdb

• [x] Demographic Parity.popularity.0.01.100.fa-ir.ndkl.dblp

• [x] Demographic Parity.popularity.0.01.100.fa-ir.skew.dblp

• [x] Demographic Parity.gender.0.01.100.fa-ir.ndkl.imdb

• [x] Demographic Parity.gender.0.01.100.fa-ir.skew.imdb

• [x] Demographic Parity.gender.0.01.100.fa-ir.ndkl.dblp

• [x] Demographic Parity.gender.0.01.100.fa-ir.skew.dblp #########################################################################

• [x] Demographic Parity.Popularity.Greedy.IMDB ( due 4/28/2023)

• [x] Demographic Parity.Popularity.GreedyConservative.IMDB ( due 4/28/2023)_

• [x] Demographic Parity.Popularity.GreedyRelaxed.IMDB ( due 4/28/2023)_

• [x] Equality of Opportunity.Popularity.Greedy.IMDB ( due 4/28/2023)

• [x] Equality of Opportunity.Popularity.Greedy_Conservative.IMDB

• [x] Equality of Opportunity.Popularity.GreedyRelaxed.IMDB ( due 4/28/2023)_

• [x] Demographic Parity.Gender.Greedy.IMDB

• [x] Demographic Parity.Gender.Greedy_Conservative.IMDB

• [x] Demographic Parity.Gender.Greedy_Relaxed.IMDB

• [x] Demographic Parity.Popularity.Greedy.DBLP

• [x] Demographic Parity.Popularity.Greedy_Conservative.DBLP

• [x] Demographic Parity.Popularity.Greedy_Relaxed.DBLP

• [x] Equality of Opportunity.Popularity.Greedy.DBLP

• [x] Equality of Opportunity.Popularity.Greedy_Conservative.DBLP

• [x] Equality of Opportunity.Popularity.Greedy_Relaxed.DBLP

• [x] averaging methods for fairness outputs such as skew and ndkl for different folds

• [x] Full paper writeup for ECIR ir for good

• [x] Resource paper writeup

• [x] Fair Top-k Ranking with multiple protected groups

• [x] Add FA*IR: A Fair Top-k Ranking Algorithm to the pipeline #80

• [x] Add another fairness metric e.g. skew (or from normalized discounted difference (rND), ratio (rRD), and KL-divergence (rKL) proposed in Measuring Fairness in Ranked Outputs)

• [x] FA*IR: A Fair Top-k Ranking Algorithm ( due 4/28/2023)

• [x] Clean and push equality of opportunity function ( due 4/28/2023)

• [x] Inferring the gender labels for DBLP dataset #36

• [x] Inferring the gender labels for IMDB dataset #36

• [x] Collect and process GitHub dataset

• [x] Explore papers for other state-of-the-art score-based ranking and learning to rank methods

[hosseinfani]

@Hamedloghmani Any progress with any of these tasks?

[Hamedloghmani]

@hosseinfani Yes, these are the stuff that are going to be reported by the end of the day:

• Summary of FA*IR: A Fair Top-k Ranking Algorithm will be finished and uploaded
• Our labeling procedure for DBLP has 3 phases : 1) Analysis and automatically extract unique names from DBLB 2) API purchase and label unique names 3)Patch back the names to the dataset ( log is in #36 ) Phase 1 has finished last night we are on phase 2, I will purchase the API today.
• The write-up of the summary of deterministic reranking algorithms will be finished and uploaded
• Analysis and unique name results for the whole DBLP dataset is available in Teams -> Adila -> DBLP Labeling files
• Github crawler is on 800,000 record. About 200,000 to go

These are next in the queue:

1. Adding another fairness metric ( my choice is normalized discounted difference (rND) )
2. Finishing gender labeling for DBLP
3. adding dblp to the pipeline

[hosseinfani]

@Hamedloghmani how about equality of odds implementation?

[Hamedloghmani]

@hosseinfani My understanding from demographic parity, equality of opportunity and equalized odds is that if we want to include equalized odds or equality of opportunity, we have to somehow enter a label to our problem to define whether an expert is 'qualified' or not. And then examine how qualified/non-qualified experts place into the team. I think ( with my knowledge at this point, I might be wrong) these are not suitable for our problem and demographic parity which falls under group fairness criteria is more helpful.

My personal suggestion is going with Group/Individual fairness criteria instead of the 3 that I mentioned at the beginning since most of the significant work in fairness did the same.

I would be happy to hear your input on this. I can dedicate more time to equalized odds if you believe it would be helpful for us.

[hosseinfani]

@Hamedloghmani I update the tasks. Please come up with a timeline in this issue page (don't create another doc/issue). Also, drop by during office hour every week to discuss your progress.

[hosseinfani]

@Hamedloghmani pls keep this issue page updated with your fall23 plan.

[Hamedloghmani]

@hosseinfani Sure, the page has been successfully updated with the tasks that we discussed and you kindly mentioned in our Excel sheet.