Balance response took ~7 mins

[lippytak]

From our loving user Francesca:

Hey team, I just texted an EBT card # to balance and it responded with the typical “Gracias!” (Which, btw, should have the “¡” preceding it) message, but it took 7 minutes for it to text back. Just thought you guys might want to know. Happy Tuesday J

@daguar let's look into this today, though I wouldn't say this requires an immediate fix unless it's happening in all cases. I'll do some manual testing now and report back here.

[lippytak]

Eeep, my first try took ~40 mins. I'll bet this has to do with the volume of the start of the month. @daguar let's break and discuss this now.

[lippytak]

@daguar is looking into Twilio logs. I am going to:

• try manual call to make sure CA line is working right
• check incoming logs to see how users are reacting

[lippytak]

CA line is working as usual.

[daguar]

Dev appears to be working with normal latency (one attempt from me).

[daguar]

Twilio status board says things are all good: http://status.twilio.com/

[daguar]

It doesn't appear that queuing is the issue (Twilio logs say no queue time on responses).

[lippytak]

Still seeing intermittent problems, even on a separate subaccount. I texted 8 EBT numbers in quick succession and only got 6 responses:

Hmmm....

[lippytak]

Seems intermittent now and equally affecting all phone numbers, so we're thinking this is a Twilio side issue. Still looking it but consider this officially downgraded from EBF-status.

[daguar]

We now have confirmation that there is a Twilio-level issue happening — will report more as we find out.

[lippytak]

@daguar I just updated the metrics data as of 2:25pm and glanced through the ~300 balance checks from today. As always there are a few imperfect experience (see below) but I didn't see a single instance of someone complaining about the delay. So I think we're :+1: and no need to alert users. Feel free to look through yourself and LMK if anything jumps out at you.

[lippytak]

BTW the above example is another good case for https://github.com/codeforamerica/balance/issues/272

Also I'm going to message that user personally from our textit.in account right now.

[lippytak]

[lippytak]

Via our one-of-a-kind Twilio dev evangelist:

Hey all, Torkel is deploying a fix as we speak. Your transcriptions should be working properly in the very near future. Really appreciate you tracking this one down for us. I think y'all are are hooked up on credits through Twilio.org, but if you need more dropped in your account in the future, or if y'all need some Twilio t-shirts, let me know.

Just tested with 10 simultaneous balance checks and got back LIGHTNING FAST (~60 seconds) responses:

Looks like this issue is :+1:

[lippytak]

(oops wrong screenshot...edited!)