codeforamerica / balance

A text message system for checking one's EBT card balance (SNAP benefits and more)
MIT License
47 stars 37 forks source link

Balance ops maintenance #283

Closed daguar closed 9 years ago

daguar commented 9 years ago

This is a ticket for just taking a general look at logs and errors over the past 1-3 months and do some quick fixes as is useful.

Some potential things:

daguar commented 9 years ago

We've got some ghost errors (in Twilio as "Error: 11205 - HTTP connection failure") where the server simply isn't responding to a request. It's hard to diagnose. Might be connectivity issues on Twilio's end; could be Unicorn, or some other weirdness (like the odd cycling behavior New Relic is causing).

It's fairly infrequent (1-2 times a month on average). Here's the distribution of such errors over the past few months:

screen shot 2015-04-06 at 11 11 23 am

daguar commented 9 years ago

Added "Remove New Relic" — we're not using it, and it's causing some weird errors.

daguar commented 9 years ago

Added clarification to above note that the ghost error type in Twilio is Error: 11205 - HTTP connection failure

daguar commented 9 years ago

Have reviewed errors in logs. Interesting finding — most of the recent 500s are from the actual message where a user has texted "stop". This is because they are added to the blocklist immediately, and so the app tries to respond but cannot and raises 500.

daguar commented 9 years ago

Nothing pressing here, so closing this issue, noting that the remaining issues can be taken on when there's spare time.