502 issue: Schedule and Execute Twilio Status Updates

[k-macmillan]

User Story - Business Need

For any messages updates that failed to reach our servers we want to make sure we still update the status of those notifications. We wrote a function to call Twilio and update the notification with the new status, this is the ticket to find notifications that need that little update and execute it.

• [x] Ticket is understood, and QA has been contacted (if the ticket has a QA label).

User Story(ies)

As a VA Notify client I want my notification status current So that I know if my notification was delivered

Additional Info and Resources

• Celery crontab schedule

Acceptance Criteria

• [x] Celery beat crontab used to schedule the task
• [x] Task executes every 5 minutes
• [x] Task queries the notifications table for records with a Twilio provider that are not in a final state and were updated > 1 hour ago, limit 500
• [x] All notifications returned from the query are updated OR logged in the event an update did not happen (such as a 429/rate limit response)
• [x] This work is added to the sprint review slide deck (key win bullet point and demo slide)

QA Considerations

The only way for us to e2e test this is to in some way kill our ability to receive or process the notification from Twilio. We can adjust the lambda handler code to just return and see if the cron picks it up an hour later. Using a twilio test number, it may be beneficial to test this once just to ensure it works, then fire off 500 to make sure our system can handle it and Twilio doesn't rate limit us or something. Twilio's rate limiting is not clear and most searches point to the "verify" service.

Potential Dependencies

Updating old notifications in the notification_history table.

[npmartin-oddball]

Hey team! Please add your planning poker estimate with Zenhub @cris-oddball @EvanParish @k-macmillan @MackHalliday @mchlwellman

[mchlwellman]

I have the celery task coded and it seems to be working. Today I have to do testing and get ready for review.

[mchlwellman]

I have tested this in dev, by sending a SMS, updating the notification in the db and letting the job update. I've added some exception handling for 429s to break out of the job, because they are all going to fail. I need to add a test and then open the PR.

[cris-oddball]

PR approved and merged, deployed to Perf for testing.

QA PASSED

• [x] TC-01: A single message sent to Twilio to my number. First, set the project-perf-delivery-status-processor-lambda to do an immediate return. Sent a Postman SMS vis Twilio at ~1:40PM MDT. "id": "193f7aeb-266a-4c39-a543-42fa01aabe35" in service_id e024930e-a56d-45dc-ba96-7c34929db954. Observed that the message was delivered to my phone and the GET v2/notifications/:notification_id route returned "status": "sending". Wait until around 2:40 MDT to see if the cron kicked off and processed the status correctly (as 'delivered')

Logs showing the cron job kicking off, finding the one status to check, making the call and then updating it

• [x] TC02: Fire off 500 SMS sent via Twilio. Leave the handler to return early, wait an hour to see if the cron picks up and starts calling Twilio for updates. See if any 429s result. Sent them at 2:20 PM MDT. Will collect ids and wait until 3:20 PM MDT.

Logs supporting successful updates - 500 updates expected

500 updates processed with 200 statuses (no other http status seen)

Noting most of those messages were updated to

"status": "permanent-failure", "status_reason": "Unable to deliver",

Perhaps because this was an AWS simulated number? Only 63 were delivered.

On the next check, 0 notifications were found to update.

Redeploying lambda.

Notes:

• observed that the cron fires off every 5 minutes (log: Getting notifications to update status)

[cris-oddball]

Leaving this open to remind myself to send it to prod Tuesday morning.

[cris-oddball]

On it's way to Staging.

[cris-oddball]

Handed off to Evan to send to Prod (first pipeline to auto-post release notes).

[cris-oddball]

On prod, closing ticket.