Add traffic lights to support identity checks

[tompollard]

At the moment, reviewing identity/credentialing applications involves subjective decisions based on loosely defined rules. This has various problems, such as:

• Slow, draining work for people reviewing applications.
• Unclear process for applicants.
• Inconsistent decisions.

We could consider adding a traffic light next to each identity/credentialing submission in the admin console. The idea would be that the color (e.g. red, amber, green) would help reviewers to make an objective decision.

• Green: likely good to approve
• Amber: needs manual review (and/or contact reference)
• Red: needs more information

We would begin by identifying characteristics of applications that can be verified as true/false by a function. We would then use these checks to determine whether or not the identity meets our requirement for approval, e.g.: the applicant has

• .edu email domain linked to their profile
• ORCID linked to their profile
• @xxxx.xxxx consortium email domain linked to their profile?
• NIH commons account linked to their profile
• UToronto account linked to their profile
• etc

Flag application as green (~most requirements met), amber (~few requirements met), red (~no requirements met).

[tompollard]

@lamawmouk do you have any thoughts on this?

[lamawmouk]

Thanks @tompollard the traffic idea sound interesting for the review process. I think adding some of the list of characteristics that aren't available on the submission page to the user e.g. NIH commons account and character max limit to research topic section etc. would be enable us to flag applications upon review.

[lamawmouk]

Workflow/pipeline suggestions:

Green:

• Users who provide an academic email/institutional email.

Orange:

• Reference with an academic email
• Users provide a webpage link (e.g. google scholar, papers, github.io websites ). If none of these are available use references papers (provide a DOI link).
• Detailed summary topic description.

Red:

• No institutional email/ academic email
• No webpage links
• Incomplete fields e.g. institution location
• One line summary description or one word description.

[tompollard]

Thanks Lama, this is helpful! How do you suggest we aggregate these items to come up with a single recommendation for an application? For example, are you saying that for "Orange" the user must check all of the following boxes?

• Reference with an academic email
• Users provide a webpage link (e.g. google scholar, papers, github.io websites ). If none of these are available use references papers (provide a DOI link).
• Detailed summary topic description.

If someone has yes to "Reference with an academic email." but a "One line summary description.", should we mark the application as orange or red?

[lamawmouk]

Yes for each color we would match the criteria with it. We could chat later to add or remove some of these but above are my initial thoughts.

Yes that is a good point about the overlap, I think there should not be an overlap in the color criteria. Therefore, we should put a mini word limit on the research summary in this case. So if someone has yes to "Reference with an academic email." They should have a detailed summary and we mark the application as orange.

I have created other issues # on git to help solidify these criteria check boxes and perhaps might add the mini word limit too as PR.

[tompollard]

From discussion with Lama, I think the idea is that we will identify various trust markers (e.g. has_institutional_email; has_credentialed_reference; has_linked_orcid; etc) and then assign based on all, any, none.

Green = all(has_institutional_email, has_credentialed_reference, has_linked_orcid)
Orange = any(has_institutional_email, has_credentialed_reference, has_linked_orcid)
Red = none(has_institutional_email, has_credentialed_reference, has_linked_orcid)

Initially the traffic light will just provide a visual indication for the recommended response. Green: Approve; Orange: Conduct manual search; Red: Request more information.

[lamawmouk]

From discussion with Michael, below is our proposed workflow:

Green (likely to approve): Users who provide an academic email/institutional email and ORCID

Orange: (needs manual review (and/or contact reference) 1) Users provide third party email domains--> reference check (contact reference who already has an academic/institutional email)

2) Users provide a webpage link (e.g. google scholar, papers, github.io websites, if none of these are available use references papers (provide a DOI link)---> manual check for approval

To implement 2) rule, we need to require users to add a webpage link upon submitting the credentialing application as suggested here #1773.

Red (needs more information): None of the above--> request more information e.g. webpage links to papers/google scholar/ academic webpage with profile

@tompollard , how do these initial rules sound like for flagging the credentialing workflow?

[tompollard]

thanks @lamawmouk those sounds like a great start to me!

[lamawmouk]

Thanks @tompollard for merging, hopefully this helps when the semester starts! For the next step I would like to send an automated message for the rejection (red light status), without prompting the admin to write a description. The options currently in the rejection email include:

• It was incomplete, or included obviously incorrect information (perhaps as a result of browser auto-fill).
• You are a student, postdoc, intern, or trainee, but did not list your supervisor (a faculty member or someone with a senior research appointment at your institution) as reference.
• Your research summary did not include sufficient information, or was in some other way inadequate.

I would like to add to that list:

• Provide a webpage link e.g. google scholar, paper links, personal webpages, academic profile webpages, LinkedIn, GitHub.

How does that sound?

[tompollard]

For the next step I would like to send an automated message for the rejection (red light status), without prompting the admin to write a description.

I don't think there should be an automated rejection email in this case. If the application is clearly incomplete, it should not have been submitted. Perhaps think more about validation rules for the identity form.

[tompollard]

Hmm, perhaps you are suggesting that the response form should be automatically populated with relevant text? This sounds fine with me.

[lamawmouk]

Sorry, I mean the system currently has the admin write text for rejection and lists other options of why it might have been rejected in the email sent to the user. It includes these: It was incomplete, or included obviously incorrect information (perhaps as a result of browser auto-fill). You are a student, postdoc, intern, or trainee, but did not list your supervisor (a faculty member or someone with a senior research appointment at your institution) as reference. Your research summary did not include sufficient information, or was in some other way inadequate. https://github.com/MIT-LCP/physionet-build/blob/dev/physionet-django/notification/templates/notification/email/process_credential_complete.html I would like to added to this current list and have it sent as a notification, instead of having the admin type the reason for the rejection.