Filter out flagged participants as "toIgnore" from QC report for a given rule

[jacobmpeters]

Tasks:

• [x] Filter participants flagged as toIgnore=true in the rules file if the toIgnore column exists. The participant should be flagged only for the specific rule. All other rules should be applied for that participant.
• [x] Add a "Rule ID" column to the rules file, so that each rule can be uniquely identified for this filtering process.
• [ ] Add a separate "sheet" to the report (an Excel sheet) for filtered rules/participants.
• [ ] Add a separate "sheet" to the report for unfiltered rules/participants.
• [ ] Update the qc_rules_recruitment.xlsx with the current the "live" rules file from box: rules file from box. You could use boxr to automate this if you'd like. We will need to do this at some point anyway.
• [ ] When done. Please add some concise documentation for this feature to the README file for the repo.
• [ ] Finally, share updates with Madhuri and Kelsey, who will be using the new toIgnore filter.

*The most recent QC report was uploaded to Box and Madhuri added true or false to the toIgnore column to indicate which participants she would like to remove for which rules.

*Please make a "handle_toIgnore_flag" branch for this task.

@rebexxxxxx please work on this task as part of the qaqc knowlege transfer. Eventually we would like you to be one of the "go to" people for making updates to our qaqc process.

We should try to complete this by Friday 12/1 if it does not conflict with higher priority tasks.

[jacobmpeters]

[jacobmpeters]

@rebexxxxxx Do you think you might have time to work on this next week? If not, I might tackle it

[jacobmpeters]

@rebexxxxxx I was chatting with @mnataraj92 and it would make her life a lot easier if we can finish this issue. Should we set up a meeting early next week to check in on this? We can discuss what still needs to be done and if it's easier for me to take over this issue, I can do that.

[rebexxxxxx]

@mnataraj92 i am so sorry i didnt mean to ignore this! totally my fault its taken this long. @jacobmpeters and i are going to regroup on monday and start working on the solution.

[rebexxxxxx]

hi @jacobmpeters I figured out the column combination necessary to identify unique rule_id and labels within the qc_rules_recruitment.xlsx file. The combo of: "ValidValues" and "Notes from QC process" will demarcate rule_ids that have identical label values. Attached here is a spreadsheet identifying the repeated labels, the number of times the label is repeated and for each repeated label i have included the data corresponding to those observations in separate sheets which are labeled by label value. please let me know what the next steps are to complete this issue. updated_repeated_labels_data.xlsx

[jacobmpeters]

Thanks @rebexxxxxx!

Next steps:

• [x] Using what you've learned above, add a column the this version of the QC Report called rule_id column that I can use to map to the rules file.
• [x] Write a function that ingests the updated report file file and generates a CSV with token and rule_id fields that indicates which tokens for a given rule for which toIgnore==TRUE.

I can take it from there and modify the qaqc.R script itself to use this information to filter out the individuals.