Large number of Push requests shut down server - can the push adapter be stress-tested?

[cfloisand]

I had a very strange issue caused by push notifications happen last week with an app I'm working on. The app lets users post community reports that show up in their feed based on which communities they have subscribed to. A special city-wide community is available to admin users, and when they post a report to this community, push notifications are sent out to everyone. In this particular case, it would have sent out approx. 1500 push notification requests.

This caused our server (hosted on heroku with 1 dyno at the time), to exceed its memory limitation and deny all subsequent requests. Additionally, users kept getting the same push notification delivered multiple times (anywhere from 10 - 20 times!). Which brings me to a few questions regarding parse-sever-push-adaper:

• Does the push adapter attempt to deliver notifications multiple times if it thinks it didn't succeed? I'm trying to account for the multiple notifications being sent to users, and I see nothing in our backend code that would spam the server with multiple push requests.

• In order to further troubleshoot this and find the cause, I would like to stress-test the push adapter and server by generating a hight number of push requests, but I see no way of doing this. Is there a way I can test sending ~1500 push requests in our dev environment using the parse server push adapter? I've tried generating a bunch of fake Installations and then sending out push notifications, but it only ends up delivering to installations with valid device tokens (understandably enough). I tried using random device tokens, and also creating multiple Installations with the same valid device token, but neither worked.

Any help and/or insight into what could have caused this problem on the parse server push adapter side would be greatly appreciated.

[flovilmart]

@cfloisand the adapter don't retry sending on it's own. How many calls to the Push API did you do ? do you have some info about the _PushStatus queries that you ran? Can you provide that please?

[cfloisand]

@flovilmart There were about 1500 calls to Parse.Push.send(). And the vast majority of the pushes have their status stuck on "Sending" (and a few on "Failed").

[flovilmart]

And all those calls targeted one and only one installation?

[cfloisand]

Each of those calls targeted one user, but if that user had more than 1 Installation, then it could have targeted more than 1 Installation. But the number of Installations in the database is very close to 1:1 with the number of users, so in most cases, it would have only targeted 1.

[flovilmart]

How did you call the parse-server, one push after the other or everything in parallel?

[cfloisand]

I think it's one after the other. Each call to Parse.Push.send() is added to a promise array, which is then returned in a Parse.Promise.when() call.

[cfloisand]

And for reference, here is the query:

sendPushToUsers: (users, data) => {
    const pushQuery = new Parse.Query(Parse.Installation)
    pushQuery.containedIn('user', users)
    return Parse.Push.send({where: pushQuery, data: data}, utilitiesLib.masterKey)
}

sendPushToUsers, in this instance, was called ~1500 times with one user in the users array. So that did make me think earlier when you brought up how many calls to the API were made that it would be better to call the API once with all 1500 users rather than the other way around. It would still be good to know if this was indeed the issue though.

[flovilmart]

So that would be in parallel. it looks something like

let promises = [];
for (let users in allUsers) {
   promises.push(sendPushToUsers(users, data)); // this make a call to the API, doesn't wait for the previous to finish
}
Promise.when(promises);

You can easily change it so promises chain:

let promise = Promise.resolve();
for (let users in allUsers) {
   promise = promise.then(() =>sendPushToUsers(users, data)); // this make a call to the API, and wait for the previous promise to finish
}
return promise;

With this, you should not take your server down.

One easy way to try, is to replace the Parse.Push.send with just a find call.

You'll see how your server behaves.

Also, did you create the proper indexes on your _Installation table so read on the user is 'fast'. You can see in the mongo logs 'slow queries'

[cfloisand]

I will try that and check the indexes on the Installation table later today or tomorrow and follow up. Thanks for your help!

[flovilmart]

you're welcome.

[cfloisand]

@flovilmart I've tested this now, and it turns out that the issue was caused by overloading the server with requests. So calling Parse.Push.send() for each user 1500 times caused a backlog of requests that bogged down the server.

In my test, I did as you suggested and replaced Parse.Push.send() with find() and executing that 1500 times resulted in a big spike in response time, memory usage, and low/failed throughput. Calling it once with 1500 users registered barely any spikes at all as it resulted in just 5 requests instead of 1500+.

Thanks again.

[flovilmart]

Thanks! Glad we could help!

_{Sent with GitHawk}