EBF: service outage (probably Twilio callback problem)

codeforamerica / balance

A text message system for checking one's EBT card balance (SNAP benefits and more)

MIT License

47 stars 37 forks source link

EBF: service outage (probably Twilio callback problem) #286

Closed daguar closed 9 years ago

daguar commented 9 years ago

Investigating

daguar commented 9 years ago

Here's what we know so far:

Billing has a balance, so not a payment issue
Our code base has not changed, so it's unlikely an issue on our end
The last successful transcription API callback call was at Apr 06 19:19:40 (so it's been non-operational ~15 hours)
Twilio status ( http://status.twilio.com/ ) shows everything up, but this was also the case last time we encountered transcription API platform issues

We're opening a ticket and also cc'ing some folks who helped last time this API had an issue.

daguar commented 9 years ago

Restarted the Heroku app — no effect
Confirmed that the most recent deploy was 8 days ago (so no production codebase change around time of issue)

daguar commented 9 years ago

@lippytak is currently putting up a "service down" auto-response for people texting in giving them the phone number they can call.

lippytak commented 9 years ago

We now auto-reply with this message for all incoming messages:

Sorry, this service isn't working right now. Please try again tomorrow. For now you can call (###) ###-#### to check your EBT balance.

(Including Spanish, bleh)

daguar commented 9 years ago

Heard back from Twilio support — it seems that they were indeed having an issue and it appears to be resolved now. (It's resolved on their end, and we have confirmed restored functionality on our end.)

Next steps:

@lippytak will turn off auto-replies and restore the link to our service for phone #s
@daguar will follow up with Twilio
@daguar will write up a postmortem

lippytak commented 9 years ago

:+1: all #s are back up and running.

@daguar please glance through the Message URLs on the Twilio #s before you close this issue.

daguar commented 9 years ago

:+1: confirmed that all production Twilio #s are pointed at the production server.

This incident is over, so I'm closing this. There are two items coming out of it:

288 Write a postmortem
287 Set up monitoring for the Twilio Transcription API callback that had issues here (and before)

codeforamerica / balance

EBF: service outage (probably Twilio callback problem) #286

288 Write a postmortem

287 Set up monitoring for the Twilio Transcription API callback that had issues here (and before)