iOS pushes partially failing for 15,000 installations

[samjarman]

Hi Team,

We're pushing to a large audience of about 15k people, a mix of android and iOS users.

Versions: OS: Ubuntu parse-server:4.5.0 @parse/push-adapter:3.4.0 @parse/node-apn:4.0.0 Node: v14.15.5

The push started off fine at 04:54:17, some rejections, mostly accepted – as expected. GMC was ticking along fine too.

At 04:54:20, we see the first error start to appear:

Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS VError: stream ended unexpectedly
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.<anonymous> (/home/USER/prod/node_modules/@parse/node-apn/lib/client.js:186:25)
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.emit (events.js:315:20)
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.EventEmitter.emit (domain.js:467:12)
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at endReadableNT (internal/streams/readable.js:1327:12)
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at processTicksAndRejections (internal/process/task_queues.js:80:21)
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS  APNS error transmitting to device %s with error %s 8a81d299ce83c26812b398c0f16977d795bbcecaf77bf5b8d9ab3112eb2ca4e9 VError: stream ended unexpectedly
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.<anonymous> (/home/USER/prod/node_modules/@parse/node-apn/lib/client.js:186:25)
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.emit (events.js:315:20)
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.EventEmitter.emit (domain.js:467:12)
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at endReadableNT (internal/streams/readable.js:1327:12)
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at processTicksAndRejections (internal/process/task_queues.js:80:21) {
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS   jse_shortmsg: 'stream ended unexpectedly',
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS   jse_info: {}
Feb 18 04:54:20 parse parse-server[95742]: ERR! parse-server-push-adapter APNS }

At 04:54:22, after some successful sends we see a new error.

Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS VError: apn write timeout
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.<anonymous> (/home/USER/prod/node_modules/@parse/node-apn/lib/client.js:207:34)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at Object.onceWrapper (events.js:421:28)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.emit (events.js:315:20)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.EventEmitter.emit (domain.js:467:12)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at callTimeout (internal/http2/core.js:2259:8)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream._onTimeout (internal/http2/core.js:1921:5)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at listOnTimeout (internal/timers.js:554:17)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at processTimers (internal/timers.js:497:7)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS  APNS error transmitting to device %s with error %s ecf6e5bf4481d278722684c7033b96db69b43c484eb8719bb41466a9b1e95e64 VError: apn write timeout
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.<anonymous> (/home/USER/prod/node_modules/@parse/node-apn/lib/client.js:207:34)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at Object.onceWrapper (events.js:421:28)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.emit (events.js:315:20)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.EventEmitter.emit (domain.js:467:12)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at callTimeout (internal/http2/core.js:2259:8)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream._onTimeout (internal/http2/core.js:1921:5)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at listOnTimeout (internal/timers.js:554:17)
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS     at processTimers (internal/timers.js:497:7) {
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS   jse_shortmsg: 'apn write timeout',
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS   jse_info: {}
Feb 18 04:54:22 parse parse-server[95742]: ERR! parse-server-push-adapter APNS }

And we see no successful pushes after that.

Our config JSON is (with obvious redactions):

{
    "port": 1337,
    "appId": "[REDACTED]",
    "masterKey": "[REDACTED]",
    "clientKey": "[REDACTED]",
    "javascriptKey": "[REDACTED]",
    "restAPIKey": "[REDACTED]",
    "databaseURI": "mongodb://[REDACTED]",

    "cloud": "dist/cloud/main.js",
    "publicServerURL": "[REDACTED]/api",
    "appName": "[REDACTED]",
    "allowClientClassCreation": false,
    "sessionLength": 315619200,
    "expireInactiveSessions": false,
    "revokeSessionOnPasswordReset": false,
    "emailAdapter": {
        "module": "@parse/simple-mailgun-adapter",
        "options": {
            "fromAddress": "noreply@[REDACTED]",
            "domain": "[REDACTED],
            "apiKey": "[REDACTED]"
        }
    },
    "push": {
        "android": {
            "apiKey": "[REDACTED]"
        },
        "ios": {
            "token": {
                "key": "[REDACTED].p8",
                "keyId": "[REDACTED]",
                "teamId": "[REDACTED]"
            },
            "topic": "[REDACTED]",
            "connectionRetryLimit": 10,
            "production": true
        }
    }
}

We had better reliability until a few days ago when pushed stopped working completely for iOS. We upgraded to the versions above in response to that and saw success again with small push volumes.

Any ideas would be very welcome for anything we can do to improve reliability? If you need any more info about our system I can provide that, too.

Thanks! Sam

[davimacedo]

How is your deployment scenario? I am wondering if the process is failing when sending all these pushes.

[samjarman]

If I've understood your question correctly, We're running with systemctl

[Unit]
Description=Parse Server

[Service]
WorkingDirectory=/home/USER/prod
ExecStart=/usr/local/bin/node dist/index.js parse-server-prod-config.json
Restart=always
# Restart service after 3 seconds if service crashes:
RestartSec=3
KillSignal=SIGINT
SyslogIdentifier=parse-server
User=[REDACTED]
Environment=VERBOSE_PARSE_SERVER_PUSH_ADAPTER=1

[Install]
WantedBy=multi-user.target

I saw no crash in the logs or server restarts. The VM we're on has 3.5GB of data and 1 vCPU, running on Azure's VMS - running Linux (ubuntu 18.04)

[davimacedo]

You can try to increase the timeout limit by passing requestTimeout option (ms, default 5000) to the ios setup. In some point you might want also to use multiple processes to send the pushes in order to continue scaling.

[samjarman]

@davimacedo Thanks, we'll give that[using a more generous time out] a go!

[davimacedo]

Let me know then.

[samjarman]

Hey @davimacedo

We just did another push to our userbase and we saw an increase in successful pushes from ~5k to about ~13k of the ~15k. An improvement for sure but not perfect.

We're still seeing a lot of the below errors. Any further ideas?

Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS VError: stream ended unexpectedly
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.<anonymous> (/home/USER/prod/node_modules/@parse/node-apn/lib/client.js:186:25)
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.emit (events.js:315:20)
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.EventEmitter.emit (domain.js:467:12)
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS     at endReadableNT (internal/streams/readable.js:1327:12)
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS     at processTicksAndRejections (internal/process/task_queues.js:80:21)
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS  APNS error transmitting to device %s with error %s 99d3a58efffa8ecc5a39b03003f98a0f278caf26e71b23ecf293977e6a4d066e VError: stream ended unexpectedly
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.<anonymous> (/home/USER/prod/node_modules/@parse/node-apn/lib/client.js:186:25)
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.emit (events.js:315:20)
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS     at ClientHttp2Stream.EventEmitter.emit (domain.js:467:12)
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS     at endReadableNT (internal/streams/readable.js:1327:12)
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS     at processTicksAndRejections (internal/process/task_queues.js:80:21) {
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS   jse_shortmsg: 'stream ended unexpectedly',
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS   jse_info: {}
Feb 22 04:30:50 parse parse-server[42961]: ERR! parse-server-push-adapter APNS }

[davimacedo]

Have you checked your server metrics? I am wondering if you maybe do not have enough bandwidth to send all those pushes at once. Were you able to successfully send 15K pushes with older versions?

[mman]

Just to add my .2 cents, I have started to see these couple times a day after upgrading from 4.4.0 to 4.5.0, but what is interesting is that I have not found any actual change in push adapter around that upgrade time.

[samjarman]

@davimacedo That's correct, we were able to send 15k pushes before we updated versions.

Previous versions: @parse/node-apn: 3.1.0 @parse/push-adapter: 3.2.0 parse-server: 4.3.0 Node: 8.x (Unsure of what version we were on)

We updated all these versions because iOS push sudden stopped working (with the same APNS VError: stream ended unexpectedly) and we suspected this change was pushed early.

Our server metrics look fine to me (not a particularly trained eye though). Our CPU doesn't go above 60% when the push happens and network traffic doesn't seem to hit a ceiling. We're using a VM in Azure, which I assume has a high limit for network traffic.

[funkenstrahlen]

I have seen the same error when sending push notifications on my production deployment. It started on 29.02.2021. Therefore I thinks its related to this change on Apple's side: https://developer.apple.com/support/wwdr-intermediate-certificate

Instead of using certificates I migrated my backend to use APNS auth key mechanism to send pushes to APNS: https://developer.clevertap.com/docs/how-to-create-an-ios-apns-auth-key

After using APNS auth keys for push notifications all errors were gone and it works fine.

This is how the push config looks like when using apns auth key:

push: {
  ios: {
    token: {
      key: '/file/path/to/AuthKey_XXXXXXXXXX.p8',
      keyId: "XXXXXXXXXX",
      teamId: "YYYYYYYYYY" // The Team ID for your developer account
    },
    topic: 'com.domain.appname', // The bundle identifier associated with your app
    production: false
  }
}

However I see @samjarman already uses apns auth keys?

[mtrezza]

I'm closing this as it does not seem to be a Parse Server Push Adapter issue.

• For help with Parse Platform we recommend our community forum
• For coding questions we recommend Stack Overflow using the parse-platform tag
• For network and server questions we recommend ServerFault using the parse-server tag

Feel free to comment if you have any questions and we can re-open this issue.