PROMIS Rollout to Backlog

[sonyekere]

• Placeholder for requirements Release timeline of issue will be further discussed once 'capacity of the system to handle people logging into MyConnect and answering a survey in large numbers all at once' is addressed. @anthonypetersen @Davinkjohnson

[brotzmanmj]

Thanks for getting this started. Adding requirements here:

PROMIS/QOL survey needs to be sent to participants who were verified prior to December 1, 2023.

1) Will need a new series of notifications for PROMIS/QOL Backlog (Hana)

2) Need to expand existing trigger for PROMIS/QOL to trigger off the 1st contact attempt in this new notification schema in addition to the current trigger off the existing PROMIS/QOL 1st contact attempt that is in place now. (DevOps task)

3) Need to decide what capacity to aim for in terms of the number of people in the backlog we send this to at once. (Davin/Tony discussing).

4) Once we have this decision to (3), we will vary the range of verification dates that we use in the conditionality of the notification.

5) Revise metrics and add some new metrics to track the response rate. (Kelsey)

[anthonypetersen]

Based on Google's documentation... Firestore, Load Balancers, and Cloud Functions all seem to be capable of handling thousands of connections simultaneously.

[brotzmanmj]

Adding @anthonypetersen for implementation thoughts on this

[anthonypetersen]

Goal is to de-couple availability of survey from first notification and move to our "typical" approach. Would need to do the following:

• Update display logic for PROMIS survey to just look at how long participant has been verified(DevOps)
• Revert default value for PROMIS Survey Status to Not Yet Started, update value for existing participants(DevOps)
• Remove logic in notifications to watch for first notification(DevOps)
• Remove notification spec condition referencing 914594314(Hana)
• Rely on Primary Field to control email not going out until X days after verification date 914594314(Hana)

Doing the above should allow us to not need a new notification spec.

Let me know if you have any questions / concerns @HanaShiho @brotzmanmj

[brotzmanmj]

@anthonypetersen Will we still have a 'not yet eligible' status for people who are less than 90 days post verification? We rely on that for metrics purposes

[anthonypetersen]

@brotzmanmj can metrics be adjusted to just calculate what that variable is doing? (is verification date more than 90 days?) From a dev-ops perspective this variable would become basically useless and once we de-couple PROMIS logic from the first notification the variable will not be updated until the next time the participant logs into the PWA, so it won't be as accurate. Adjusting the calculation of metrics to not rely on that variable would provide a much more real-time look.

[brotzmanmj]

@KELSEYDOWLING7 or @rebexxxxxx can you advise on Tony's comment above? This would impact the coding for the PROMIS metrics in the weekly operations report.

[KELSEYDOWLING7]

@brotzmanmj Right now the PROMIS Completion vs Notifications Report doesn't actually use the eligibility variable. It only pulls participants with at least one '3mo QOL Survey Reminders'.

For the Operations report, 'Not eligible' is set if SrvQOL_3moStatus_v1r0 is 'Not yet eligible'

[anthonypetersen]

@KELSEYDOWLING7 can the operations report be adjusted to have Not Eligible set if verification date isn't more than 90 days old?

[KELSEYDOWLING7]

@anthonypetersen It can be. Will we still have a 'Not started' status if they are eligible? Because the 'Not yet eligible' is basically set by default.

dataqol$qolcomplt <- ifelse(dataqol$d_320303124 == 231311385 ,"Submitted", ifelse(dataqol$d_320303124 == 615768760, "Started", ifelse(dataqol$d_320303124 == 972455046 , "Not Started", "Not Eligible")))

[anthonypetersen]

@KELSEYDOWLING7 in pseudo code:

IF 320303124 = 231311385 -> "Submitted" ELSE IF 320303124 = 615768760 -> "Started" ELSE IF 320303124 = 972455046 IF Current Date - Verification Date > 90 -> "Not Started" ELSE -> "Not Eligible"

[KELSEYDOWLING7]

@anthonypetersen I think the codes are the same. I confirmed there's no one with verification less than 90 days ago with a 'Not Started' status.

But there are currently almost 21,000 with status 'Not Eligible' that have been verified for 90 days or more. If I change the 'Not Started' code to either d_320303124 == 972455046 OR Current Date - Verification Date > 90, I get this. Which has significantly more 'Not Started' but eligible participants.

@brotzmanmj Do these numbers seem correct to you? This is the table with the older code for reference:

[anthonypetersen]

@KELSEYDOWLING7 @brotzmanmj I'm guessing this is because at the moment we've only enabled this survey for participants who joined in December 2023 or later

[KELSEYDOWLING7]

@anthonypetersen Right, ok ya that makes sense. @brotzmanmj If this looks good for the upcoming changes, I can make these updates in the next automation update

[brotzmanmj]

@KELSEYDOWLING7 This change won't go into effect until the July release on July 31st so please hold off on the changes to the report/automation until then

[brotzmanmj]

@KELSEYDOWLING7 and @anthonypetersen Because the PROMIS survey has been based on the notification specs, there are some other conditions on that email that influence the current counts of whether someone is eligible for it or not. @HanaShiho please enter those here. The are things like excluding withdrawn ppts, deceased ppts, and those who refused future activities.

When we change away from the notification trigger, we should account for those as well

[HanaShiho]

PROMIS/QOL survey invitation for verified participants who meet all of the following criteria: have not withdrawn consent, have not refused all future activities, have not refused all future surveys, and is not deceased.

[brotzmanmj]

A couple of further thoughts upon reflection: 1) We automatically block all future surveys for people after they withdraw so I think this is no need to address this specifically in the conditionality for the PROMIS survey to appear. 2) Though we don't send notifications about surveys to people who refuse future activities or surveys, we do not typically block the surveys from showing up on their ToDo list on MyConnect. They can either choose to do them or not. So perhaps no special code needed for this either. We do not send them notifications asking them to complete activities they have refused. 3) Deceased participants we assume will not log in to MyConnect so there is nothing in particular we do to block surveys from MyConnect for them. Again, we just won't send them notifications.

About the notifications: @anthonypetersen We will still need a second series of notifications to the backlog. If we remove the 'verified >= December 1, 2023' criteria as we discussed but keep the timing of the existing messages at verification + 90 days, 93 days, 99 days, the backlog of participants will get all the notifications at once.

[anthonypetersen]

@brotzmanmj what if you just removed the verified >= December line from the notification specs one at a time, and did them the correct amount of days from each other?

[brotzmanmj]

@anthonypetersen I don't think that would work for the prospective participants who will each be at a different day post-verification

[brotzmanmj]

Update: Tony and I discussed and we agree we can remove the verified >= December criteria from the messages one at a time. Remove it from Contact 1 on Day 0 defined as the day of the release when we push the new trigger to prod. Remove the condition from Contact 3 three days later. Remove it from Contact 2 nine days later. Ditto for the last contact attempt.

[KELSEYDOWLING7]

@brotzmanmj I think I may need more clarification on how to proceed. Can we set up a meeting next week to discuss?

This code was very complicated with the before/after 3pm verification times, how to find and position participants who didn't receive notifications when we had expected them to, ect. Table 1's denominator is a separate code from table 2, but if we have increasing eligibility we'll have to figure out exactly how to define those additional people. Also Verified >=December is the criteria for the entire report, not a criterial for a table or row. Finally, this may be difficult to test without having data to verify correct counts on. Is this being pushed to DEV or Stage at all first?

This report runs every Monday. With day 0 being Wednesday 7/31, and Day 3 being Saturday 8/1, the report wouldn't have to change until the Monday 8/5 run. Day 9 would be 8/9, so we'd need to update it again for the 8/12 run. Luckily this gives us a bit of time.

[brotzmanmj]

Hi @KELSEYDOWLING7 , I'm out next week Mon-Weds but we have more time because this is not going in the July release. It will be part of the August release at the end of August. So no changes should be made until the end of August. I plan to pull together the requirements into a single document since there has been so. much back and forth in this github issue and will share that out and then we can meet.

[KELSEYDOWLING7]

@brotzmanmj Great!!

[sonyekere]

Following the meeting held today, August 1st, the PROMIS rollout will be prioritized for the August release. @rohanjay10, could you please work with Hana to pull together and finalize the PROMIS rollout requirements? Please give it a first pass, and @brotzmanmj will confirm accuracy once it's done.

[rohanjay10]

@sonyekere Sure, will do

[rohanjay10]

Hi all, here is the latest copy of the requirements that we reviewed in our meeting today. Couple things still need to be confirmed (per the comments)- @anthonypetersen and @KELSEYDOWLING7 please review this when you get the chance.

[rohanjay10]

@jhflorey The Backlog change requirements document above is updated. The changes for this 'backlog' August release would be merged with the original PROMIS survey(requirements here)

Currently the PROMIS/QOL Survey: • Available to participants who were verified >= 90 days on or after December 1, 2023(and not withdrawn, not deceased etc.). • The trigger is the first contact notification(SMS email) that is sent to the participant 90 days after they are verified.

The August Backlog expansion goal: • Allow the survey to also be made available to those participants who are verified >= 90 days before December 1, 2023 in the backlog. When they log in to the PWA the PROMIS survey is available. • Trigger the survey based on the participant logging into the PWA and having the survey available to them if they are verified >= 90 days instead of the trigger being the first contact notification.

Implementation

With updates per the meeting and discussion with Tony

Goal is to de-couple availability of survey from first notification and move to our "typical" approach. Would need to do the following:

• Update display logic for PROMIS survey to just look at how long participant has been verified(DevOps)

• Logic in the code needs to be updated to look at how long the participant is verified

• If a participant is verified >= 90 days and they log in to myConnect the survey will get triggered Verification Status(821247024) should = 'verified'

• Revert default value for PROMIS Survey Status to Not Yet Started, update value for existing participants(DevOps)

• The plan is to update the default value to 'Not Started' from 'Not yet eligible' for all participants in our database

• @mnataraj92 to change this survey status default value in dictionary and then @jhflorey to change in code

• Tony thinks this value change to 'Not yet started' should happen after release, prioritized for verified participants.

• Remove logic in notifications to watch for first notification(DevOps)

• Remove notification spec condition referencing 914594314(Hana)

• The same set of notification specs will be used and just edited to remove ‘verified >= December 1,2023’ criteria after each contact on Day 0, 3, 9, 45 and check after each day with a dry run.
  • Day 0 of release being when new trigger pushed to prod.

• Rely on Primary Field to control email not going out until X days after verification date 914594314(Hana)
• Rely on Primary field to control email notification not going out until 90 days (or more) after verification date (primary field will remain date of verification; concept ID 914594314 and set number of days = 90).

Doing the above should allow us to not need a new notification spec.

Testing

Hana put together a draft of the test plan which Madhuri and I will review.

@brotzmanmj @anthonypetersen @HanaShiho Feel free to add on or make changes.

[jhflorey]

@mnataraj92 @sonyekere my code change is ready. We can test on dev.

[mnataraj92]

tested in dev, ready for stage

[HanaShiho]

When we go live with the PROMIS backlog, we will remove the 12/1/23 cut-off from each existing PROMIS spec on the day it's scheduled to go out per the cadence. Currently, four PROMIS specs are scheduled to go out on Day 90, Day 93, Day 99, and Day 135 post verification. Assuming the prod push occurs on Wednesday night (8/28) as scheduled, we will remove the cut-off date from the 1st contact on Thursday (8/29) before 3PM. Based on that schedule, the 2nd contact will go out on Sunday (9/1) and so on. Because this would send out messages over the Labor Day weekend Michelle proposed that we adjust the existing cadence so the 2nd contact goes out on Tuesday (9/3) instead of Sunday (9/1), better enabling us to respond if there are any issues. This would also shift the following contacts by a few days to avoid the subsequent reminders going out over a weekend. Does anyone have any objections and/or suggestions regarding adjusting the cadence?

[brotzmanmj]

Discussed with Davin and he agreed with the proposal of first email on Thursday 8/29, second email on Tuesday 9/3. He suggested third email keep as planned on Saturday 9/7 to capitalize on higher engagement on the weekend which I agree is a good idea. Assuming all will be running smoothly by then.

[mnataraj92]

tested in stage, ready for prod