Push Notifications reported as sent and without error but not received in case of ~ 20 notifications

[lukasnagl]

In case we send a certain amount of push notifications, they don’t seem to be delivered correctly. Everything works correctly for a lower amount of push notifications. We do not use persistent connections.

Here’s some very rough logging code that we had in place:

    client.push(*notifications)
    Rails.logger.info "PNS Job (1/2): Send #{notifications.length} notifications. Notification Sample: #{notifications.first.inspect}. Errors: #{notifications.collect(&:error).compact}. Sent: #{notifications.collect(&:sent?)}"
    Rails.logger.info "PNS Job (2/2): #{client.devices}, Push Infos: #{push_options.inspect}"

All following examples were created with the production build of the app, using the production certificate and Houston production client with houston 2.2.1.

Here’s the log output of a working example. As you can see, the errors are empty and both notifications are reported as being sent, and they are correctly received on both clients.

PNS Job (1/2): Send 2 notifications. Notification Sample: #<Houston::Notification:0x000000070bb240 @token="<REPLACED>", @alert="Some Text", @badge=1, @sound=nil, @category=nil, @expiry=nil, @id=0, @priority=nil, @content_available=true, @custom_data={:custom_data=>{:rscs=>[{REPLACED}]}}, @sent_at=2014-09-26 10:15:13 +0200>.
Errors: []. Sent: [true, true]
PNS Job (2/2): [], Push Infos: {:alert=>"Some Text", :badge=>1, :content_available=>true, :custom_data=>{:rscs=>[{REPLACED}]}}

Here’s the log output of an example that does not work. Again, the errors are empty and all notifications are reported as being sent, but they are not in fact received on the target devices. The target devices include the two devices from the previous working example:

PNS Job (1/2): Send 23 notifications. Notification Sample: #<Houston::Notification:0x0000000c8dc488 @token="<REPLACED>", @alert="Some Alert", @badge=1, @sound=nil, @category=nil, @expiry=nil, @id=0, @priority=nil, @content_available=true, @custom_data={:custom_data=>{:rscs=>[{REPLACED}]}}, @sent_at=2014-09-29 15:33:36 +0200>.
Errors: []. Sent: [true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true]
PNS Job (2/2): [], Push Infos: {:alert=>"Some Alert", :badge=>1, :content_available=>true, :custom_data=>{:rscs=>[{REPLACED}]}}

What could be the source of the error or what else could we look into to narrow down the problem?

[micred]

See #72

[lukasnagl]

@micred Is this really the same issue? It seems to me that #72 is based on the presence of invalid tokens. The invalid tokens are retrieved in the code sample:

Rails.logger.info "PNS Job (2/2): #{client.devices}

And in both provided examples the resulting array in the logs is empty, meaning that Houston reports that there was no invalid token used.

[micred]

I don't know exactly. Try progressively remove a device token until (maybe) you find the bad token amount them and then they will start again.

[micred]

mmm... I don't know if it solve your problem but I find that adding a sleep solve my problem. I'm curious if it also solve your problem, can you try using Persistent Connections and adding sleep(0.5) between each connection.write(notification.message)?

Thank you.

[micred]

Further details: http://redth.codes/the-problem-with-apples-push-notification-ser/ Grocer similar bug: https://github.com/grocer/grocer/issues/14

[lukasnagl]

I’ve tried switching to persistent connections and the sleep like you suggested and hit issue #77, Errno::EPIPE (Broken pipe).

@micred: Would you suggest to try out https://github.com/nomad/houston/pull/73 or should I try to use http://joshsymonds.com/blog/2013/10/17/sidekiq-plus-houston-production-ready/?

[lukasnagl]

I tried using pull request https://github.com/nomad/houston/pull/73, but to no avail. Still hitting issue https://github.com/nomad/houston/issues/77.

[elado]

Same problem here. Got no error from this gem, but tried sending with https://github.com/noodlewerk/NWPusher and got invalid token error.

[kaneda]

Already commented on https://github.com/nomad/houston/issues/72 but the quick workaround is to send each notification separately

[elado]

@kaneda I do, and errors are still empty.

[kaneda]

@elado yes, it won't return a proper error, but at least the rest of your notifications won't be dropped

[sgwilym]

@kaneda Doesn't sending each notification separately mean opening a new connection for each notification?

[kaneda]

@sgwilym it does, unfortunately.

[evrenios]

any fix found for this issue, i gave a wrong token to test and i noticed that it didn't raise any error so error is still nil of the notification even tho its random characters.

[LuizEdgar]

Hello, this issue was solved? If no, how are you handling this?

[kaneda]

@LuizEdgar, not to the best of my knowledge, instead I send each notification separately to users instead of en masse.

[LuizEdgar]

Thanks @kaneda.

Well... but Apple says that it can be considered as an DoS Attack.

"Keep your connections with APNs open across multiple notifications; don’t repeatedly open and close connections. APNs treats rapid connection and disconnection as a denial-of-service attack. You should leave a connection open unless you know it will be idle for an extended period of time—for example, if you only send notifications to your users once a day it is ok to use a new connection each day."

How bad can it be? How many notifications are you sending per day?

[kaneda]

@LuizEdgar it could be considered a DoS attack. I'm not sending many, in the hundreds.