nightscout / cgm-remote-monitor

nightscout web monitor
GNU Affero General Public License v3.0
2.39k stars 71.64k forks source link

Failed APNs Attempt is crashing Nightscout instance #8291

Open AndyLow91 opened 6 days ago

AndyLow91 commented 6 days ago

I've been investigating a problem with Nightscout/Loop Caregiver/Remote Commands since yesterday, and I've now found some interesting information about the cause.

It's easy to say that it's an Apple issue, and I'm now pretty sure it is, but also, the issue with Apple is also crashing folks Nightscout sites, so there should be an update done in Nightscout Core to prevent this crash from happening.

For those of you interested:... I've had users report they see a 502 Error: Bad Gateway. This error means that your Nightscout site is unreachable, and shows whenever your site is rebooting. Looking at the current and previous logs of one particular user, I see this for the reason it crashed:

APNs delivery failed: [
  {
    device: '9a7e87c696325caeda085b06cdee05c324ae402c31ca185d92e8a60c2f9773a2',
    error: VError: stream ended unexpectedly with status null and empty body
    at ClientHttp2Stream.<anonymous> (/opt/app/node_modules/@parse/node-apn/lib/client.js:241:27)
    at ClientHttp2Stream.emit (node:events:527:28)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
      jse_shortmsg: 'stream ended unexpectedly with status null and empty body',
      jse_info: {},
      cause: [Function: ve_cause]
    }
  }
]
/opt/app/lib/server/loop.js:138
completion("APNs delivery failed: " + response.failed[0].response.reason);
^
TypeError: Cannot read properties of undefined (reading 'reason')
at /opt/app/lib/server/loop.js:138:75

To further test this theory, (and you can test this yourself too) when I open up a terminal, and try to ping either api.push.apple.com or api.sandbox.push.apple.com (the two endpoints where APNs messages are sent to, depending if you build by xCode or Browser build) I sometimes get a response, and sometimes the request just times out.

To me this confirms that Apple's APNs servers may be overloaded with traffic at these times and are unable to respond. I'd be interested to hear if anyone else could test this theory or has any other input to give on the situation?

AndyLow91 commented 6 days ago

PR #8292 to resolve this fatal crash and instead, display an error to the user.