search

appleboy / gorush

A push notification server written in Go (Golang).
MIT License
7.85k stars 831 forks source link

Question: TooManyProviderTokenUpdates from APNS #512

Open Tillman-Z opened 4 years ago

Tillman-Z commented 4 years ago

Hi there!

I just have witnessed the TooManyProviderTokenUpdates error response from APNS. This would indicate that the derived key (which should be renewed every 20-59mins as far as I understand Apple's recommendation) would be renewed too quickly. Is this a config error on my part or is this an issue with GoRush?

Best, Tillman

slimus commented 4 years ago

Hi @Tillman-Z !

429 The server received too many requests for the same device token

Can you please describe your task? Why are you sending a lot of pushes to one device token?

Tillman-Z commented 4 years ago

Thanks for your quick response @slimus! I am not sending a lot of pushes to one device token. And if I understand it correctly then the error message refers to updating the provider token (which is derived from the p8 token) too often = earlier than 20mins. Best, Tillman

slimus commented 4 years ago

Sorry, I copied incorrect info :( I read this articles: https://forums.developer.apple.com/thread/86288 https://github.com/matthijs2704/vapor-apns/issues/60 But I think this is not your case So, just to clarify: you are using p8 and send pushes to different tokens. Is it correct? How many pushes did you send before you get this error?

Tillman-Z commented 4 years ago

This is a production system and it's sending out about 1 push per second. I have several instances of goRush (6) all running on the same server. Those instances are 3 iOS (belonging to different teams with different p8 tokens) and 3 FCM instances. I am not seeing this error often and it also is apparently recovering from it automatically. But every now and then a push message will not get through with this error response.

slimus commented 4 years ago

Thank you for information. I think we should check part with p8 token and reusing connection. Issue from library which used in gorush: https://github.com/sideshow/apns2/issues/132

Tillman-Z commented 4 years ago

Thanks so much for your fast responses! I would appreciate it if you updated this thread once you have committed a (potential) fix. :)

Best, Tillman

pyrho commented 4 years ago

Hey, I'm really not an expert in any of the subjects I'm about to talk about, but I was asking myself how the token was refreshed, here is my investigation report.

According to apple's documentation, the token needs to be refreshed every 59m (but no more than every 20m).

The call to client = apns2.NewClient(certificate).Production() creates the client/token and appears to be only called by gorush upon startup (in main, via InitAPNSClient). According to apns2's doc:

// NewTokenClient returns a new Client with an underlying http.Client configured
// with the correct APNs HTTP/2 transport settings. It does not connect to the APNs
// until the first Notification is sent via the Push method.
//
// As per the Apple APNs Provider API, you should keep a handle on this client
// so that you can keep your connections with APNs open across multiple
// notifications; don’t repeatedly open and close connections. APNs treats rapid
// connection and disconnection as a denial-of-service attack.

And this seems to be the case here.

I have not found other bits of code messing with this client (but once again, I don't know how to Go); so I'm guessing the role of refreshing the client is left to the apns2 lib. And it seems to be the case, see this code from apns2. Prior to pushing the notification, apns2 will check if the token is more than 3000s seconds (50 minutes) old (go Time.unix() returns a timestamp in seconds, see here, and refresh the token automatically).

So my guess for your bug is that (assuming you only have one auth key), every 50 minutes(ish) your 5 gorush instances try to refresh the same token, resulting in the error you're seeing.

I guess fixing this for multiple instance would require gorush to either share the token among instances, or to be able to control the token refresh behavior within apns2.

Tillman-Z commented 4 years ago

Good thinking...but no ;-) Those instances all have different TeamID / KeyIDs. So it seems that apns2 sporadically does recalc a key too often. Although I fail to see how that's possible looking at the go code (but I am not a go person either).