SPIKE: RemoteDisconnected Error

department-of-veterans-affairs / notification-api

Notification API

MIT License

16 stars 8 forks source link

SPIKE: RemoteDisconnected Error #1019

Open k-macmillan opened 1 year ago

k-macmillan commented 1 year ago

Issue

We are seeing Remote end closed connection without response in our logs. Investigate and find a resolution.

Note: This exception leads to a retry after 5 minutes. It has been verified that these are making it through eventually.

Timebox

A couple days

Checklist

[ ] Determine next steps and tickets needed

Additional Info/Resources

https://www.twilio.com/docs/api/errors
Search for "Remote end closed connection without response" "message" in cloudwatch (with quotes)
Alternative search: "Twilio send SMS request" "failed" "message"
Can possibly be tested locally with locust

mjones-oddball commented 1 year ago

Hey team! Please add your planning poker estimate with Zenhub @cris-oddball @EvanParish @ianperera @jakehova @k-macmillan @kalbfled @ldraney

mjones-oddball commented 1 year ago

Please add your planning poker estimate with Zenhub @justaskdavidb2

ianperera commented 1 year ago

FYI https://www.twilio.com/docs/api/errors

mjones-oddball commented 1 year ago

Jacob is assigned because he feels with his personal twilio account he may be able to troubleshoot and figure this out. If needed we can talk to VEText about using their support contacts to better understand why these errors are popping up in their twilio logs.

k-macmillan commented 2 weeks ago

I believe this may be a gunicorn error. I was able to replicate it while performance testing another app and fix it by setting the gunicorn timeout to infinite. Infinite timeout is not feasible for a production system though, so additional investigation is still required.