UChicago Incentive Metrics

[kmazzilli]

Requestor(s): Kaitlyn Mazzilli

Table or Figure title: edits to Tables 37, 38, and 39. We would like to add a new table that will now be Table 40.

Metrics to be updated or created: Weekly Operations Report

Variables needed (include variable name and Concept ID): I believe Madhuri can help with these

Notes: Due to the change in UChicago incentive options, we would like to make edits to/double check the construction behind three existing tables and add one more table to the report. As a reminder, UC was only offering a cash incentive and recently started to offer the NORC gift card incentive.

Table 37:

• Please remove the footnote.
• The contents of this table may not need to change, but the assumptions underlying how it's constructed should be checked. @KELSEYDOWLING7 please see the following note from Michelle. I believe we discussed this (or something very similar) on our call with Madhuri:
• "It should not assume that all Chicago participants are only getting incentives issued by Chicago. I think there is one variable for Payment Issued that both NORC and Chicago are sending data into, so how will Kelsey know who issued the incentive? The only way might be to look at the Payment Chosen variable, those that are 'cash' will be Chicago' and everything else will be from NORC. Alternatively, this might be easier...  NORC is sending back a case number (cid 320023644) with the incentive issued data and Chicago is not sending that, so the presence of a case number could be use to identify NORC Issue Incentive and the absence of that would be a 'Chicago Issued Incentive'."

Table 38:

• We would like to add University of Chicago to this table.
• Please remove the existing footnote, and add a new footnote stating “University of Chicago started offering NORC incentives on 12/14/2023”.

Table 39:

• The time for the NORC incentive for Chicago participants will now be in Table 38. With that decision, Table 39 needs to only reflect incentives issued directly by Chicago. Kelsey could you check the programming to make sure this table is capturing only Chicago issued incentives? As long as that is the case, the table can stay as it is.
• However, the footnote will need to be updated to say "UChicago is currently offering cash payment and NORC giftcards. This table depicts cash incentives only. Due to the cash incentives being provided to participants in clinic, there may be a short period of time where the incentive was provided to the participant at the end of the visit but the blood tubes had not been scanned into the biospecimen dashboard yet."

NEW TABLE TO BE ADDED:

• This can be titled "Table 40. University of Chicago Participants' Incentive Choice"
• We want this table to show counts of UC participants and whether they chose gift card vs cash. It would look just like table 37 but restricted only to UC pts where the NORC line would show UC pts that chose the NORC incentive option and the following line can say "University of Chicago Medicine - Cash".

Approved by: Michelle Brotzman

[KELSEYDOWLING7]

@kmazzilli We do have one UC participant Connect ID with both a NORC case number and cash Payment Chosen. Do we have more information on that participant? It's the only one, but I want to make sure that wasn't the UC participant that received the first UC giftcard

Also for Table 38 do we need the Total row? It doesn't seem to do much good other then the Total Number of Incentives. The other values really don't make sense...

[brotzmanmj]

That seems not expected, about that one participant. @KELSEYDOWLING7 Can you send the Connect ID?

[KELSEYDOWLING7]

@brotzmanmj @kmazzilli Oh sorry I didn't realize the CID didn't past. That was Connect_ID 2021911600 Payment Chosen "Cash" and NORC case number CS0015979

[robertsamm]

Hi @KELSEYDOWLING7 we heard back from Jaime at UC last night who shared they rolled out offering the gift card incentive on 12/7 (not 12/14 as previously thought).

She shared the following UC pts have deferred their cash payment: 1479364285 3102444927 1715408526 6363149606

[KELSEYDOWLING7]

@robertsamm Thanks Amelia! It looks like they're not among the ones who received gift cards:

7248843156 1886629139 4781783848   8862512851   1893386706 4726914725 3586434651 2532157209 7430437721 1870810858 9726355843

@kmazzilli Here are the requested table updates

[robertsamm]

@brotzmanmj @kmazzilli Oh sorry I didn't realize the CID didn't past. That was Connect_ID 2021911600 Payment Chosen "Cash" and NORC case number CS0015979

@KELSEYDOWLING7 for this one, I agree with Michelle this does seem odd. It seems like they might have gotten both a cash and gift card payment? Do we have a QC check for this already, if not we should add one. And @FrogGirl1123 how do you suggest we look into this one? Should we send to both UC and NORC and ask them whether they were issued an incentive from both sites?

[KELSEYDOWLING7]

@mnataraj92 Are the incentives part of the recruitment QC? If so, do we have a rule to catch any participants with both the NORC case number and "Cash" payment chosen?

[mnataraj92]

@KELSEYDOWLING7 no we don't have a rule to catch pts with both the NORC case # and the "cash" payment chosen. Should it just be that if one is there the other should not be? (and vice versa)

[KELSEYDOWLING7]

@robertsamm Do you mind confirming?

@mnataraj92 My understanding was that if there's a NORC case number, cash shouldn't be chosen; of cash was chosen, there shouldn't be a NORC case number. Because the NORC case numbers should only be for gift cards

[robertsamm]

Thanks @KELSEYDOWLING7 and @mnataraj92 that is my understanding as well!

[mnataraj92]

@robertsamm that sounds good! I'll add that to the QC rules @jacobmpeters

[KELSEYDOWLING7]

@kmazzilli Are those tables above ready for acceptance or do they need to be edited at all?

[brotzmanmj]

@KELSEYDOWLING7 I'm looking into the case from last week where the Connect ID 2021911600 had a cash incentive noted but also a NORC incentive as that should not have occurred. I want to make sure there is not an issue with the derivation. The SMDB gives me only limited visibility. I can only see that they became incentive eligible on 11/10/23 and were issued their incentive on 11/10/23. You said the incenitve chosen was 'cash' so that would have been given by the Chicago team. And then they have a case number which is our only indication that perhaps NORC also issued an incentive. Is that accurate? I think what we need to do is ask NORC to look up this case number and see if/when they issued an incentive.

[KELSEYDOWLING7]

@brotzmanmj That is correct, and the case number is CS0015979 if you need it

[brotzmanmj]

Thanks. Madhuri is reaching out to NORC and Chicago to find out more. For the metrics, something is off either in the data or the titles. It might be better to hop on a call to go over them.

[KELSEYDOWLING7]

@brotzmanmj I did some digging, of those that refused their incentives, 174 still had a NORC case number and 26 were blank instead of null.

Also, when I had the logic as (Has a NORC case number =NORC, Payment Chosen is cash = UC, Otherwise = Other) that all of those 'Other' category participants were the ones that had no incentive issued or refused. Not sure what to make of that.

[KELSEYDOWLING7]

If I update the conditionality to exclude those who had Payments Chosen as Cash, there are only two people that received an incentive without a case number, but we have none in that incentive not issues or refused category

[brotzmanmj]

Are the 26 who refused incentive and have NORC case number blank all UChicago site participants? And, I think something is still not making sense because the Total Incentive Eligible is 12,500 in one table version above and 9,992 in the other. This number should be easily knowable and we should establish what it is and then work backward from it to make sure the other counts add up to it. Can you a frequency count of the variable for Incentive Eligible (CID 731498909) with no other conditions?

[KELSEYDOWLING7]

@brotzmanmj Yes, those 26 are UC participants. The two tables have two different conditionalities. The first is just that they're incentive eligible, the second table has the conditionality that they have to be incentive eligible but did not choose Cash as a Payment. So the 12,500 in the first table includes all eligible

[brotzmanmj]

OK so now we know that NORC sends back a case number for all incentives that are issued or refused. For Chicago participants who refuse incentive, they have a blank case number. So our rule based on existence of a case number = NORC and everyone else who has incentive issued = yes or refused = yes and no case number is Chicago. The 26 should be in the row above where it is in the top table

[brotzmanmj]

I'm not sure what's going on with the 2 that had an incentive issued but we don't know from whom. The 679 just represent people who are incentive eligible and have not been issued or refused an incentive yet. When we change the table to the new specs that Kaitlyn will provide, it will make more sense as to where those go.

[KELSEYDOWLING7]

By the statification it looks like those 26 are from UC but did not take cash. That's why they're in the 'Other' category.

In case its needed, the 2 with incentives and no case number are CIDs 2670278092 and 4694236727

[brotzmanmj]

Thanks, I'll look up those two in the SMDB and see what I can deduce. For the 26, they don't classify cleanly the way the table is constructed, so we'll change the table format to what Kaitlyn provides.

[brotzmanmj]

2670278092 is a KPHI participant, so their incentive could only have been issued by NORC. According to the SMDB it was issued on 3/2/2023.

4694236727 is a HP participant, so also had to have been issued incentive by NORC. It was issued on 2/23/2023.

@mnataraj92 can you first check the prod manual change log and see if we did something manually to the incentive data for either of these participants? That might explain the lack of a case number. If that doesn't explain it, then please ask NORC to check their records for these two and see if they can explain.

For the 679, I think it would be worthwhile to share the list of tokens for those with NORC and ask them to confirm against their records that those are individuals who have not been issued or received an incentive but are somewhere in their system and are in process. Kelsey can you generate a list and provide to Madhuri, and Madhuri can you share it on the Box location where we share lists with NORC and ask them to check? It is not urgent, can be done after the holiday.

[kmazzilli]

Hi everyone - below are the notes and edits from our call today. I have attached these notes with mock up tables for Table 37 and 40 here: UC Incentives Metrics Mock ups.docx

Table 37: • We would like to edit this table to reflect data for everyone that is not a UC participant. • The title can read “Baseline Incentive Status for NORC Incentive Eligible Participants (All Sites Except UChicago)”

Table 38: • need to fix how this is being produced to limit the table to incentives issued by NORC. If possible use the presence of a case # for the records included in this table. If incentive refusals also get a case number, then limit by the combination of presence of a case # and incentive issues = yes. • Kelsey to check if there are case #s associated with refusals; then table 38 for UC should reflect NORC incentives only • Please change the existing footnote to reflect the correct date “University of Chicago started offering NORC incentives on 12/7/2023”.

Table 39: • This should reflect data for cash incentives only • The title can say “Table 39. Time From Incentive Eligible to Cash Incentive Issued by UChicago in Days”

New Table 40: • This should reflect the incentive type given to UC participants, the denominator is everyone whose site = Chicago and Baseline Incentive Eligible = Yes. • This can be titled "Table 40. Baseline Incentive Status for Chicago Incentive Eligible Participants"

[KELSEYDOWLING7]

@mnataraj92 Please see attached 679 Tokens.csv

[mnataraj92]

@brotzmanmj @KELSEYDOWLING7 Connect IDs 2670278092 and 4694236727 are both in the prod manual change log and we manually updated baseline incentive chosen for both of these individuals.

I'll reach out to NORC regarding the list of tokens Kelsey generated.

[brotzmanmj]

Thanks, that solves a mystery! Can you please ask NORC to provide us with the Case Numbers for those two Connect IDs so we can add that in a prod manual change? (In the future, now we know that if we do a prod manual change to update the incentive issued, we need to also include the case number if it is a NORC incentive). @KELSEYDOWLING7 , this means you should be able to use the existence of a NORC case number to to distinguish NORC incentives (issued or refused) from Chicago-provided incentives, the exception being if they have a case number and a 'cash' incentive, we need to reflect those in the report and also in the QC so they can be investigated.

[mnataraj92]

sent!

[KELSEYDOWLING7]

@kmazzilli @brotzmanmj

[brotzmanmj]

Hi @kelsey, can you make the changes that are in the document Kaitlyn attached? She has mock-ups of the actual tables there.

[KELSEYDOWLING7]

@brotzmanmj @kmazzilli Table 38 and 39 should already match the request.

Table 37 updated below

For Table 40, as far as the percentages are concerned, are you looking for something like this?

[brotzmanmj]

Hi @KELSEYDOWLING7 That looks like a cumulative percent. Can we just get the column percent? And the label of the column, instead of Incentive Issued can it be Incentive Status? And the total Incentive Eligible at the bottom.

[KELSEYDOWLING7]

@brotzmanmj Ah I see what you mean. But the NORC Giftcards and Cash payments will add up to the total incentives issued. So you would have the incentives issued total counted twice, once in the total and once through adding the cash to the giftcards.

We would need to remove the requested 'Total Incentives Issued' row and just have Gift card, cash, refused, and incentive not issued or refused.

Also this is what I currently have in the report, I just ran the one above in R to add the percentages to the incentive types

[brotzmanmj]

@KELSEYDOWLING7 I see what you mean. I just updated the MockUp for Table 40 in Kaitlyn's document. Does that solve the issue?

[KELSEYDOWLING7]

@brotzmanmj I'm not seeing any update from the mockup she posted above. Do you mind posting the updated one again?

[brotzmanmj]

UC.Incentives.Metrics.Mock.ups.docx

[KELSEYDOWLING7]

@brotzmanmj Thank you, please let me know how this looks

[brotzmanmj]

great!

[KELSEYDOWLING7]

@brotzmanmj With the new table 36 in #96 this will become table 41. Is that ok?

[brotzmanmj]

Sure, that's fine

[KELSEYDOWLING7]

@brotzmanmj Noticed that Tables 38 (now Table 39) and table 39 (now table 40) had the wrong counts. Just fixed.

[brotzmanmj]

Hi @KELSEYDOWLING7 something still seems off. Where did the 679 participants go from the previous run who were incentive eligible but has not been issued or refused an incentive?

[KELSEYDOWLING7]

@brotzmanmj Based on the updated request mock up those were removed. Table 37 used to be all participants, including UC, where those participants were coming from. The markup requested that Table 27 be updated to everyone that is not a UC participant and not falling into the 'Other' category, so they were removed.

[brotzmanmj]

@KELSEYDOWLING7 So if they were all UC participants, they should show up on the new Table 41 but I'm only seeing 45 there.

[KELSEYDOWLING7]

@brotzmanmj I apologize, I misspoke. Those are NORC participants. There are 682 total incentive eligible participants in the cohort that have neither refused nor been issued an incentive, 637 are NORC participants and 45 are UC participants.

I had removed the 'Other' row from Table 38 (previously Table 37) to match the markup, but that row contained those participants. I was able to fix the code all together; here is the corrected Table 38

[brotzmanmj]

Awesome, thank you! Then I think we are good to go.

[KELSEYDOWLING7]

@brotzmanmj @kmazzilli Updated on the 1/8 Operations report

[kmazzilli]

@KELSEYDOWLING7 the new tables and the numbers look good.

1. It doesn't look like the footnote for Table 40 got updated to say: "UChicago is currently offering cash payment and NORC giftcards. This table depicts cash incentives only. Due to the cash incentives being provided to participants in clinic, there may be a short period of time where the incentive was provided to the participant at the end of the visit but the blood tubes had not been scanned into the biospecimen dashboard yet."

2. Would it be possible for Table 41 to get moved up to be right after Table 40 and before Figure 6.0?

Thanks!

[KELSEYDOWLING7]

@kmazzilli I'm sorry! I thought I commented on this last week. Yes, that will be updated for Monday's report