Broken push notifications

[cfabio]

Some days ago push notifications stopped working for me, this happen on multiple devices with different accounts. XEP-0357 is active and push notifications are also enabled in Chatsecure but still I receive no push notifications.

[gerroon]

Is this related? I am getting this line in my ejabberd log. It does not work for me either.

'''jabberd_s2s_in:handle_auth_failure:205 (tls|<0.539.0>) Failed inbound s2s EXTERNAL authentication pubsub.chatsecure.org -> MYDOMAIN'''

[cfabio]

I don't think so, at least not in my case since I never had that error in my logs.

[gerroon]

Btw I am not sure how push should work with Chatsecure on Ios. I definetely cant get messages if the app is killed but I can get messages if the app is in the background.

I see this message (same as above) when the app is killed

jabberd_s2s_in:handle_auth_failure:205 (tls|<0.539.0>) Failed inbound s2s EXTERNAL authentication pubsub.chatsecure.org -> MYDOMAIN

I see this message in Ejabberd log if the app is in the background

2018-05-27 13:13:19.316 [info] <0.542.0>@mod_push:enable:308 Enabling push notifications for USER@MYDOMAIN/USER-chatsecure Looking at this I am not sure if it is working or not, it looks like it kind of works?

[gerroon]

@cfabio

I will open a new bug

[jenda122]

I can confirm push notifications stopped working (they worked maybe two months ago, and I think I have not changed anything in my setup except for upgrading CS from 4.2.1 to 4.3.0). Does <iq type="result"> necessarily mean that the pubsub server is okay with the request?

Jun 30 03:44:21 c2s2345e20  debug   Received[c2s]: <message id='purple1b9769a7' type='chat' to='i@mydomain.cz'>
Jun 30 03:44:21 c2s2345e20  debug   Archiving stanza: <message id='purple1b9769a7' type='chat' to='i@mydomain.cz' from='j@mydomain.cz/bar'>
Jun 30 03:44:21 mydomain.cz:mam debug   Archiving stanza: <message id='purple1b9769a7' type='chat' to='i@mydomain.cz' from='j@mydomain.cz/bar'>
Jun 30 03:44:21 mydomain.cz:carbons debug   Skip carbons for offline user
Jun 30 03:44:21 mydomain.cz:cloud_notify    debug   Invoking cloud handle_notify_request() for offline stanza
Jun 30 03:44:21 mydomain.cz:cloud_notify    debug   Sending push notification for i@mydomain.cz to pubsub.chatsecure.org (760724E4-5B48-4B74-90BF-CC96F1200B6F)
Jun 30 03:44:21 stanzarouter    debug   Routing to remote...
Jun 30 03:44:21 s2sout247dd60   debug   going to send stanza to pubsub.chatsecure.org from mydomain.cz
Jun 30 03:44:21 s2sout247dd60   debug   sending: <iq type='set' to='pubsub.chatsecure.org' from='mydomain.cz' id='1d6e6c6a8e7c0eae2a673975e87d43b4007599579d0cddf45779282106fa4370'>
Jun 30 03:44:21 s2sout247dd60   debug   stanza sent over s2sout
Jun 30 03:44:21 c2s2345e20  debug   Received[c2s]: <message id='purple1b9769a8' type='chat' to='i@mydomain.cz'>
Jun 30 03:44:21 c2s2345e20  debug   Not archiving stanza: <message id='purple1b9769a8' type='chat' to='i@mydomain.cz' from='j@mydomain.cz/bar'> (empty when stripped)
Jun 30 03:44:21 mydomain.cz:carbons debug   Skip carbons for offline user
Jun 30 03:44:21 mydomain.cz:cloud_notify    debug   Invoking cloud handle_notify_request() for offline stanza
Jun 30 03:44:21 mydomain.cz:cloud_notify    debug   Sending push notification for i@mydomain.cz to pubsub.chatsecure.org (760724E4-5B48-4B74-90BF-CC96F1200B6F)
Jun 30 03:44:21 stanzarouter    debug   Routing to remote...
Jun 30 03:44:21 s2sout247dd60   debug   going to send stanza to pubsub.chatsecure.org from mydomain.cz
Jun 30 03:44:21 s2sout247dd60   debug   sending: <iq type='set' to='pubsub.chatsecure.org' from='mydomain.cz' id='1d6e6c6a8e7c0eae2a673975e87d43b4007599579d0cddf45779282106fa4370'>
Jun 30 03:44:21 s2sout247dd60   debug   stanza sent over s2sout
Jun 30 03:44:21 s2sin24c2180    debug   Received[s2sin]: <iq id='1d6e6c6a8e7c0eae2a673975e87d43b4007599579d0cddf45779282106fa4370' type='result' to='mydomain.cz' from='pubsub.chatsecure.org'>
Jun 30 03:44:22 c2s2345e20  debug   Received[c2s]: <iq id='purple1b9769a9' type='get'>
Jun 30 03:44:22 s2sin24c2180    debug   Received[s2sin]: <iq id='1d6e6c6a8e7c0eae2a673975e87d43b4007599579d0cddf45779282106fa4370' type='result' to='mydomain.cz' from='pubsub.chatsecure.org'>

...and the phone does not wake up

[zuglufttier]

Try testing with conversations.im to make sure it's not a problem on your server.

[jenda122]

I think conversations.im is Android-only and I don't have a phone with GApps to test it on...

However, push works with IM+ (with their MITM server), so it's definitely not my carrier's issue.

And I do get a reply from Chatsecure server - and there is presumably no error, because if I try for example push on an expired handle, I get "Discarding unhandled error iq".

So some input from CS's pubsub server admin would be great...

[gerroon]

I never have any issues with Conversations with my server, all messages are always delivered regardless if the app is open or not, Chatsecure cant seem to get messages if it is not open at least most of the times.

[jenda122]

Because you are talking about Conversations on Android, which allows the applications to run in the background (including open connections), so it works even without push notifications (at the expense of battery drain). iOS kills apps after several minutes, so push notifications are a must.

Chatsecure cant seem to get messages if it is not open at least most of the times

This really used to work (basically out of the box with Prosody and mod_cloud_notify), but stopped some time during May or June.

[licaon-kter]

@gerroon Apple has different push conditions. On my server (ejabberd) at least, users get the messages if they did not actually swipe the app. That's the only condition afaics.

@jh-cg For Prosody you also have mod_mam, mod_smacks, mod_csi and others loaded too, right?

[jenda122]

Yes, all of them. But at least now I'm getting error iq from pubsub.chatsecure.org :).

Which ejabberd module do you use? push_keepalive?

[licaon-kter]

@jh-cg conversations.im runs on ejabberd, but there is no "Android only XMPP server", it can't be. 🙄

I've got:

mod_mam:
  assume_mam_usage: true
mod_push: {}
mod_push_keepalive: {}
mod_stream_mgmt:
  resend_on_timeout: if_offline

[zuglufttier]

@jh-cg, I mean that you should try to create an account on the server conversations.im! It's a very up-to-date server. The first six months of usage are free, so you can just try it.

With my ejabberd, my only chatsecure user may even swipe the app away and notifications still work.

[jenda122]

Oh, sorry, I thought you were talking about conversations.im client.

I have tested it with a freshly installed ejabberd and it works, so the problem really seems to be in my prosody.

[licaon-kter]

@jh-cg conversations.im is a server which can be used on any client, including ChatSecure.

Conversations is an Android app that can connect to any server, including your own.

Not sure why you keep saying otherwise.

[gerroon]

@licaon-kter

Well I have the saem issue as yours which is that the users can get the messages only when their app is not closed in Ios. I am using the latest Ejabberd with proper CA directed in the config etc.

However @zuglufttier says that it should work even when the app is closed andstill gets the messages via notifications.

[licaon-kter]

In theory yes, in practice the user closes the app at night, opens it sometimes, gets god knows which or how many messages (but a effing lot of notifications).

[Echolon]

Hi,

just started an EtherCalc Pad to collect information about your and others experience about push. Feel free to add your data or additional columns (suggest only to add columns where the information can be given from a normal user)

https://ethercalc.org/29iyo1nunqdp

Cheers

[irb]

I'm not sure it's related, or new; but I have a ChatSecure user on iOS who has been bombarded with new message notifications after upgrading ejabberd to 18.12. Previously his push notifications worked fine, and push to Android clients like Conversations is also working. I don't see anything helpful in the ejabberd log. The client reports that push support (XEP-0357) is unavailable. I don't have ready access to the device unfortunately.

[chrisballinger]

@irb I wonder if there is a bug in ejabberd 18.12 causing significantly more push notifications to be sent out. There has been a massive spike in traffic to the Push API server in the last month.

[licaon-kter]

@chrisballinger 18.12 was published 5 days ago, not sure if anyone (besides some that build from git) even has it.

Maybe your app is suddenly more used hence the spike? :)

[irb]

@licaon-kter It's already out in Arch Linux.

FWIW, we uninstalled and reinstalled ChatSecure on his phone but the problem persisted. He was running CS against an older version of ejabberd successfully. The notifications come at a rate of perhaps 2/sec indefinitely. That might account for a traffic spike, but it seems to be in response to a single message received (MUC or individual, doesn't matter) by the client, not outgoing.

[chrisballinger]

I can't control when an external XMPP server triggers a push message. It seems like ejabberd is abusing the "high priority" pushes for events that should result in no user-facing message.

[licaon-kter]

@chrisballinger There was no change regarding the priority, not sure one can do that server side (care to share how?).

Again... Either you saw spikes in the last 5 days or 18.09 had this since September or 18.06 since June....so not "1 month" but spikes since 3 or 6 months ago.

Last version that touched push was 18.06 and that was an improvement for CS

/ps: I'm no dev, the above text might be 100% BS ;)

[chrisballinger]

All I know is that I somewhat recently started getting more usage tier alerts from the hosting provider. We were hitting 80-90% usage alerts in the Celery task queue towards the end of of the month for a while, but now we're only half way through December and already at 135% usage.

One problem with not collecting any analytics in the PubSub bridge is that I don't have any insight into which XMPP servers are causing the most load.