PROMIS/3-month QOL Survey Submission Report

[HanaShiho]

Please generate data tables for PROMIS/QOL survey submission following the first, second, and third email and SMS notifications; the overall structure and formatting should be similar to the baseline survey completion report.

The cadence for each notification is as follows: 1st (email and SMS): set to be triggered 90 days post verification 2nd (email and SMS): 93 days post verification 3rd (email and SMS): 99 days post verification

Note that the three PROMIS/QOL survey reminders are currently programmed to be sent to participants who were verified on or after 12/2/23, excluding backlogged participants.

[KELSEYDOWLING7]

@HanaShiho As we discussed, this includes all participants (active and passive), daylight savings is not taken into consideration and neither is the pre/post 3pm verification time lag.

PROMIS-Completion-vs-Notifications.pdf

[HanaShiho]

Kelsey also confirmed that all 167 participants in Table 2 completed the PROMIS survey the same day as the notification (not before)

[KELSEYDOWLING7]

All 167 in Table 2 and all 162 in table 3

[HanaShiho]

Correction: new pts verified on or after 12/1/23

Also as a friendly reminder, the PROMIS/QOL survey only gets triggered to people upon getting the first notification so the expected number of completed should be based on the number of people who received the first contact.

[KELSEYDOWLING7]

@HanaShiho Here's the update with the verification date pushed back. As the data is pulled for only those with PROMIS notifications, there's no issues with the denominator

PROMIS-Completion-vs-Notifications.pdf

[brotzmanmj]

@KELSEYDOWLING7 and @HanaShiho Since the emails and texts go to the same people at the same time, please rethink how to reconfigure this to represent this. One table would be best, rather than separate tables for email and SMS.

Then a separate table to indicate how many people got sent email only vs sms only vs both at each contact attempt, just based on a notifications table would be good.

There are a few footnotes that aren't accurate, please work together to fix those.

Also in the table there is no such things as completing the survey 'on or before ' contact 1, since it's physically impossible to complete it before, please update that part. You can just make it like the others, 'after contact 1, before contact 2' which would encompass completion on the day of the first contact or any day thereafter before contact 2. Thanks.

[HanaShiho]

@brotzmanmj, so @KELSEYDOWLING7 and I took another look at the report, and it looks like she already had a)verified on or after 12/1/23 and b) verified for 90 days incorporated in the code, considering PROMIS is different from any other surveys in its prospective nature. So the number of people who were eligible for the 2nd reminder correctly excluded people who met any of the exclusion criteria along with the aforementioned timing conditions.

Also, regarding your comment about some of the footnotes being inaccurate, Kelsey will update the footnotes to reflect the following :1) eligible participants will receive both emails and texts unless they opted out of receiving text messages; 2) those who refused the BL survey should still receive the PROMIS reminder; and 3) in addition to the exclusion requirements outlined for the 1st contact, the 2nd and subsequent contacts should further exclude anyone who submitted the PROMIS survey or who refused the 3-month PROMIS survey.

[brotzmanmj]

@KELSEYDOWLING7 and @HanaShiho The eligibility for the 2nd reminder needs to exclude people who are 91 or 92 days post verification because they will not be eligible for it until 93 days post verification. The eligiblity for the 3rd reminder meeds to exclude people who are 94 - 98 days post verification because they will not be eligible for it until 99 days post verification.

[brotzmanmj]

small correction to what I just posted... The eligiblity for the 3rd reminder meeds to exclude people who are 90 - 98 days post verification because they will not be eligible for it until 99 days post verification.

[KELSEYDOWLING7]

@brotzmanmj For that last update, we do already have "4 ’ Email reminder schedule: Contact 1 is sent 90 days after verification; Contact 2 on day 93 post verification; Contact 3 on day 99 post verification." that implies eligibility or lackthereof based on how long ago verification occurred. Did you want to reemphasize that in the eligibility footnote?

[brotzmanmj]

@KELSEYDOWLING7 I'm actually referring to the programming rather than the footnote. What I'm saying is, the people who are eligible for the 2nd contact attempt should not equal the number of people who were sent contact 1 minus the number of people who completed the PROMIS survey after contact 1 and before contact 2. It has to be the number of people who were sent contact 1, minus the number of people who complete the PROMIS survey after contact 1 and before contact 2 MINUS the number of people who are less than 93 days post verification. Does that make sense? This is different than the BL survey or KP biospecimen analysis we did in a similar manner before, because those were entirely from a retrospective period of time.

[KELSEYDOWLING7]

Oh! Yes ok that makes sense, and why we're seeing different numbers than expected. I'll work to fix that.

Then when I looked at all the participants there's no one that ONLY had emails or ONLY had sms. Everyone that had sms texts received emails and everyone that received emails got at least one text. But that seems to make sense as well based on the timing of the notification.

And lastly many people did PROMIS the day OF contact 1. So I adjusted that to be 'Survey Completed On Date of Contact 1', 'Survey Completed after Contact 1, before Contact 2' ect. Please let me know if that works.

[KELSEYDOWLING7]

@brotzmanmj Please let me know how this looks. PROMIS-Completion-vs-Notifications.pdf

[brotzmanmj]

@HanaShiho please take the first review and provide feedback

[HanaShiho]

@KELSEYDOWLING7, thank you for the updated report! Looks good to me, I just have minor suggestions. Can you please add a line after each contact attempt indicating how many people are ineligible and thus excluded from the denominator for the next contact attempt? (Basically, those who had not reached the necessary post-verification date.) I know you've already incorporated this into your code, but if you can list that attrition in the table, that would be great. Also, I think we can avoid separating those who completed the survey on vs. after Contact 1, but that's just my preference.

[brotzmanmj]

I agree with Hana. And I see there is still a small gap (albeit much smaller now!) between the number eligible and the number sent. My best guess is that this is due to the verification before 3pm vs after 3pm distinction. Are we taking that into account?

[HanaShiho]

Kelsey asked this question along w/ daylight saving and I told her that we wound't need to. @KELSEYDOWLING7, can you please rerun the report incorporating the verification time (before 3 PM vs. after 3 PM)? Thank you!

[brotzmanmj]

Hi @KELSEYDOWLING7 and @HanaShiho This is not about daylight savings time. This is about the fact that if verification occurs before 3pm, it counts that day as Day 0, and if verification occurs after 3pm (and up until 2:59pm) the next day, it counts the day after verification as Day 0. @KELSEYDOWLING7 this is the same way you coded the expectation in our other analysis as expected to go out on day 3, 6, 12 vs day 4, 7, 13, etc. The same coding you did then applies here as to what is 'day 90, 93, 99' for a participant. You also captured the times correctly in UTC so there was no need to worry about daylight savings. Do you recall this? Let's use the same coding here.

[KELSEYDOWLING7]

Good morning @brotzmanmj @HanaShiho . Yes, I recall that and have applied the code. Here is the update report with the first two, completion rows combined to "Survey Completed after Contact 1, before Contact 2", footnote updates, ineligible participants counted in attrition, and pre/post 3pm verification notification delays accounted for. PROMIS-Completion-vs-Notifications.pdf

My only thought was that it may make more sense to have "Eligible for Contact 3" defined as Eligible for contact 2 - finished before contact three - not eligible for contact 3 instead of Sent Contact 2 - finished before contact three - not eligible for contact 3

That would look like this below. The thought being we don't know why those 53 didn't get sent contact 2, but they aren't NOT eligible for contact 3, so they could still get contact 3

[HanaShiho]

Thanks, @KELSEYDOWLING7! For Eligible for Contact 2, shouldn't it be smaller than 1262 (i.e. 1928-584-82)? From there, more pts could have been excluded if they further met the exclusion criteria for the 2nd reminder. To your other point, it doesn't make sense to me the percentage would go over 100%, but I'm not sure about applying the eligibility criteria to Reminder 2 and Reminder 3 separately. The way I see it, the denominators (whoever is eligible for the next reminder) should get sequentially smaller, but I'll defer to @brotzmanmj.

[KELSEYDOWLING7]

@HanaShiho Thank you, you're right it should be exactly 1,262 for 'Eligible for Contact 2'. I had forgotten to apply the verification time adjustment to those rows as well.

And a percentage over 100% would mean more were sent then were eligible, which has happened in the past. If I use the original definition of 'Eligible for reminder Contact 3' as 'Sent reminder Contact 2'-'Survey Completed after Contact 2, before Contact 3' - 'Not verified long enough for Contact 3' with the corrections we still get above 100%, though very small.

[brotzmanmj]

HI @KELSEYDOWLING7 can you update this now that we have Contact 4 in prod? Thanks

[KELSEYDOWLING7]

@brotzmanmj Sure thing. How many days after verification is contact 4 sent? @HanaShiho

[HanaShiho]

@KELSEYDOWLING7 135 days post verification

[KELSEYDOWLING7]

@brotzmanmj @HanaShiho Similar to the original BL Survey Reminders, while most people verified after 3pm had their notifications delayed 1 day post verification and most people had their notifications as expected if verified before 3pm, that somehow did not apply to everyone. 24 people verified after 3pm still got their first email and/or text notification 90 days after verif instead of 91; about 6 got their second notification at day 93 instead of 94. Of those verified before 3pm, 14 received their first notif on day 91 instead of 90; 7 received their 2nd notif on day 94 instead of 93. This may account for the discrepancy between some of the eligible vs sent counts. PROMIS-Completion-vs-Notifications.pdf

[brotzmanmj]

Hi @KELSEYDOWLING7 that is interesting. Can you remind me what UTC time you're using for 3pm?

[KELSEYDOWLING7]

19

[brotzmanmj]

@KELSEYDOWLING7 Would it be possible to use the day they got their 1st contact attempt as the basis for when to expect their future contact attempts? Meaning, if they got their first contact attempt on day 90 then the next one is expected on day 93, etc. If they got their first contact attempt on day 91, then the next one is expected on day 24, etc. That would take the guesswork out of it. What do you think?

[KELSEYDOWLING7]

@brotzmanmj I should have clarified that those counts were only among those that did eventually complete PROMIS. Of those that didn't and were verified on or after 3pm:

• 13 still got their first email and/or text notification 90 days after verif instead of 91
• 12 got their second notification at day 93 instead of 94 ( the other got the second notif on time) Of those that didn't finish and were verified before 3pm:
• 8 received their first notif on day 91 instead of 90
• 8 received their 2nd notif on day 94 instead of 93

We can do that, and after reviewing it definitely gets closer to accurate timing that way. But all of the third and fourth notifications for everyone were sent on time (day 99 or 100 respectively, day 135 and 136 respectively) so that pattern wouldn't hold true all the way through if they received additional notifications.

[brotzmanmj]

@KELSEYDOWLING7 Thanks for pointing that out. I have no explanation for what is happening. My suggestion would be then to use the first notification to predict the timing of the second notification, but use the 'on time' expected days for messages 3 and 4

[KELSEYDOWLING7]

@brotzmanmj This helped, but we still have a slight overage. PROMIS-Completion-vs-Notifications.pdf

[brotzmanmj]

Hi @KELSEYDOWLING7 this is close enough and probably as good as we are going to get. We would like to have this run on a weekly basis going forward. What would be the best place to put this? Do we have a Communications Weekly Report that is internal to the CCC?

[KELSEYDOWLING7]

Hi @brotzmanmj Great. Yes, we have this folder for CCC weekly reports and I can add an additional folder for 'Weekly Notifications Reports'. Do you want this run every Monday?

[brotzmanmj]

Yes that would be great thanks

[KELSEYDOWLING7]

@brotzmanmj All set! This week's report is now in there

[brotzmanmj]

Thanks! We can close this issue then.