IOS APN (push notifications) not working with OpenFire

[ioscanner]

Is there any known issues with IOS APN when using Openfire 4.5.1? I loaded the latest plugin from the link you provided. But it seems it doesn't pickup the push notification when a message comes in. I see the push jar file loaded. I have tried with the latest version in testflight as well as a few earlier versions. So far no luck.

Thanks.

[anurodhp]

The server needs to be able to make an s2s connection to monal.im. Does your server federate?

-- Anurodh Pokharel

Sent from my iPhone

On Feb 6, 2020, at 5:32 PM, ioscanner notifications@github.com wrote:

 Is there any known issues with IOS APN when using Openfire 4.5.1? I loaded the latest plugin from the link you provided. But it seems it doesn't pickup the push notification when a message comes in. I see the push jar file loaded. I have tried with the latest version in testflight as well as a few earlier versions. So far no luck.

Thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[ioscanner]

The server is public facing. I can s2s to other servers. what server does it try to connect? I can do a test in server 2 server -> tools -> test to your server to make sure.

[anurodhp]

Try push.monal.im or ios13push.monal.im

-- Anurodh Pokharel

Sent from my iPhone

On Feb 6, 2020, at 6:04 PM, ioscanner notifications@github.com wrote:

 The server is public facing. I can so s2s to other servers. what server does it try to connect? I can do a test in server 2 server -> tools -> test

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[ioscanner]

Yes I can connect both ways to the server

[anurodhp]

When you look at notifications under settings is anything unchecked?

-- Anurodh Pokharel

Sent from my iPhone

On Feb 6, 2020, at 11:53 PM, ioscanner notifications@github.com wrote:

 Yes I can connect both ways to the server

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[ioscanner]

Everything is unchecked I don't remember what version you had that it stopped. I did report it. It has been like that I think since 4.1. You can't check or uncheck anything under notifications. It is also the same on the iPad.

[anurodhp]

Each box has an explanation of what it means and how to fix it. Every box should be checked.

[ioscanner]

They are all unchecked and you can't check them. I see the boxes, I see the info about the box, but on IOS you can't check the boxes it doesn't allow it.

[ioscanner]

Ok it looks like they have a grey check in the box, but you can't change it. They are forced. You can check and uncheck the accounts in 4.1. I rolled back to earlier builds to see. I am testing now to determine when it last worked. But so far all versions you can uncheck the notifications.

Even in 4.1 the push notifications don't work. Only once I open the Monal does it send the notification.

[ioscanner]

Hmm now I can't uncheck even the accounts in notifications. So it seems everything is checked, but still doesn't work. It looks as if everything is forced checked.

[ioscanner]

I wonder if it is a push issue with openfire apn plugin. Because when I was on Prosody it worked.

[anurodhp]

I think it may be something odd with your server or plugin

[ioscanner]

I downloaded the latest from the website you had listed. Do I need to load an earlier version? I have 0.7.0-SNAPSHOT

[anurodhp]

i actually dont know, it worked when helped test it back when i posted it but as we know software changes.

[ioscanner]

Do you know what version you tested?

[anurodhp]

I dont

[ioscanner]

I guess not. I tried to wait even after a few it doesn't push. I will load each version and see if any of them work again.

[ioscanner]

Ok I tried all versions. None work. It seems once the client reports as detached I don't get the message until I open Monal again. I tested on iPhone and iPad even 4.1-latest of Monal. I will test from another install of Openfire and verify. I even turned the firewall off on the server to ensure nothing was blocking.

[ioscanner]

Ok I tried a new install Openfire just to be sure. I can't get push to work with Openfire 4.5.1 and Monal. I have tried everything I can think of. I tried all versions of the plugin and all major versions of Monal. I did test with Prosody and it works with no issues. So it must be between your app and the push plugin. I have no way of getting in communication with the developer. They don't allow people to open tickets. I did post to the forum, but it has been many days with nothing. I am not sure if you have a way to report the issue. I had this issue before that is why I moved to Prosody. Thanks for your help.

[anurodhp]

I’ll install it an take a look. I do have contact with the devs

[ioscanner]

If you need access to a server I can get you an account and shell to test.

[anurodhp]

just ran a local test, open fire doesn't not seem to be opening s2s connections to monals push server. I dont know where the problem is but the module does appear to be buggy.

[ioscanner]

Ok, I notice something else. When viewing client sessions in openfire after I switch to a differnt app on the iPhone with monal being put to sleep by IOS. I see the client session report the IP as: Invalid session/connection. I wonder if that is causing the issue of not getting the push from Openfire.

[anurodhp]

I saw the same thing. It might be a clue

-- Anurodh Pokharel

Sent from my iPhone

On Feb 18, 2020, at 6:06 PM, ioscanner notifications@github.com wrote:

 Ok, I notice something else. When viewing client sessions in openfire after I switch to a differnt app on the iPhone with monal being put to sleep by IOS. I see the client session report the IP as: Invalid session/connection. I wonder if that is causing the issue of not getting the push from Openfire.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

[licaon-kter]

@guusdk Ping?

[ioscanner]

I did try different versions of java to check to make sure it was not a java bug that we have been hitting. I am checking to see what versions of openfire and the plugin version has worked in the past. I have never been able to get push to work. I have no issues with Prosody and IOS push. Is there anything I can try that might help you?

[guusdk]

Would you mind turning up the log level for the relevant packages, try again, and share what is being logged?

To increase the log level, edit the file lib/log4j2.xml. In that file, find this section, and add the line that I've marked here:

<Loggers>
    <!-- OF-1095: Uniform output of loading/unloading of plugins to std-out. -->
    <Logger name="org.jivesoftware.openfire.container.PluginManager">
        <AppenderRef ref="console"/>
    </Logger>
    <Logger name="org.jivesoftware.openfire.container.PluginMonitor">
        <AppenderRef ref="console"/>
    </Logger>

    <!-- OF-506: Jetty INFO messages are generally not useful. Ignore them by default. -->
    <Logger name="org.eclipse.jetty" level="warn"/>

    <!-- ADD THIS LINE -->
    <Logger name="org.igniterealtime.openfire.plugins.pushnotification" level="all"/>

    <Root level="info">
        <AppenderRef ref="all-out"/>
        <AppenderRef ref="debug-out"/>
        <AppenderRef ref="info-out"/>
        <AppenderRef ref="warn-out"/>
        <AppenderRef ref="error-out"/>
    </Root>
</Loggers>

Changes can take a few moments to be picked up, but should never take longer than a minute or so.

Next, try sending a message to a client for which push notifications should be enabled, and review the log file in log/all.log. It should list entries that relate to the "pushnotification" category.

[guusdk]

Up until moments ago, there was not a proper release of the Openfire Push Notification plugin. This required you to use a development build. These builds get updated all the time. A while ago, a breaking change was introduced, which very likely causes your issues. Just now, we've released version 0.6.0 of the Push Notification plugin. I've tested this with the developer of Monal. In our tests, push notifications were transmitted as expected. I advise you to replace your development build (0.7.0-SNAPSHOT) with release 0.6.0 (which is not a development build, and more stable).

You can download version 0.6.0 from https://www.igniterealtime.org/projects/openfire/plugins.jsp or through Openfire's admin console (you might need to remove your development build and refresh the cached list of available plugins before it pops up there).

[ioscanner]

I downloaded the latest version of the plugin and Monal, as of today. I see in the logs that it does the push just as before, but I still don't get the push on my IOS device:

2020.04.15 18:09:17 org.igniterealtime.openfire.plugins.pushnotification.PushInterceptor - Delivered a notification for user 'user' to node '4B179F90-77E4-4EE9-A7B2-5EDC78A75BBF' on service 'ios13push.monal.im'.

Do I have to open a certain port on the firewall for the push notification to be returned to openfire? I have not seen any mention of that being required.

I do still notice that openfire client sessions shows client IP as: "Invalid session/connection" after I navigate away from Monal on the iphone or ipad. Would this cause the issue of push not working?

[guusdk]

Do I have to open a certain port on the firewall for the push notification to be returned to openfire?

No. The push notification will not be sent back through Openfire at all. The flow is as follows:

1. Openfire determines that a push notification needs to be sent to you (typically when you received a new message).
2. Openfire will lookup the push notification configuration that your client (Monal) registered in Openfire. There, it finds connection details for a Monal Push Notification server.
3. Openfire instructs the Monal Push Notification server to generate a push notification for you (this is the line you see in Openfires log).
4. The Monal Push Notification server talks to a service within Apple, instructing it to send you a push notification.
5. Apple sends you a push notification directly to your phone. The happens completely out of band of XMPP. Openfire does not receive this data.

[Echolon]

@tmolitor-stud-tu

[ioscanner]

Ok, so it seems the push plugin working on the openfire side, but Monal is not pushing the notification to apple and/or apple to the phone.

[tmolitor-stud-tu]

Would be good to get a copy of the stanza sent from openfire to monal's push server. That way I can analyze if this is a bug in the push server (which I am the author of, btw.).

[ioscanner]

@tmolitor-stud-tu I can get that for you. Tell me the steps to get the stanza sent from openfire. I would be happy to provide it or even access to the server to pull what you need and test.

[tmolitor-stud-tu]

@ioscanner Thing is: I never used Openfire because its development was lagging behind when I joined the xmpp community back in 2015 (and only recently caught up somewhat).

I fear you have to ask somebody familiar with Openfire or try to find the information how to enable tanza debug logging in the docs.

[ioscanner]

I will look and see if I can find a way to dump the stanza logs.

[guusdk]

This is a stanza that was sent to Conversations push server (I've redacted the node-id and secret values):

<iq type="set" id="384-60882" to="p2.siacs.eu" from="igniterealtime.org">
   <pubsub xmlns="http://jabber.org/protocol/pubsub">
      <publish node="redacted">
         <item>
            <notification xmlns="urn:xmpp:push:0" />
         </item>
      </publish>
      <publish-options>
         <x xmlns="jabber:x:data" type="submit">
            <field var="FORM_TYPE" type="hidden">
               <value>http://jabber.org/protocol/pubsub#publish-options</value>
            </field>
            <field var="secret">
               <value>redacted</value>
            </field>
         </x>
      </publish-options>
   </pubsub>
</iq>

[guusdk]

Here's a version of the push notification that will write out the push stanza that's send to the log files: pushnotification-trace.tar.gz

You'll need to enable trace logging in, by adding a line like this in openfire/lib/log4j2.xml:

<Logger name="org.igniterealtime.openfire.plugins.pushnotification" level="all"/>

Add that near the end of the file, inside the Loggers element. There should be a similar entry that sets the log level for org.eclipse.jetty to warn. You can add it just below that line.

[tmolitor-stud-tu]

@guusdk iOS is a special plattform and apps can not be woken up in the brackground by normal (silent) pushes. Because of this an informal standard emerged which is not codified in the push xep: The push has to be a push summary and the last-message-body field has to be set for pushes that were triggered by a message stanza containing a body (or was e2e encrypted!). This way the push server can generate a non-silent push containing a general "New Message" which finally will start the app when the user opens the notification.

Example from prosody:

<iq id='id-1' type='set' from='xmpp.eightysoft.de' to='ios13push.monal.im'>
    <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <publish node='NODE'>
            <item>
                <notification xmlns='urn:xmpp:push:0'>
                    <x xmlns='jabber:x:data' type='form'>
                        <field var='FORM_TYPE' type='hidden'>
                            <value>urn:xmpp:push:summary</value>
                        </field>
                        <field var='message-count' type='text-single'>
                            <value>1</value>
                        </field>
                        <field var='pending-subscription-count' type='text-single'/>
                        <field var='last-message-sender' type='jid-single'/>
                        <field var='last-message-body' type='text-single'>
                            <value>Neue Nachricht</value>
                        </field>
                    </x>
                </notification>
            </item>
        </publish>
        <publish-options>
            <x xmlns='jabber:x:data' type='submit'>
                <field var='FORM_TYPE'>
                    <value>http://jabber.org/protocol/pubsub#publish-options</value>
                </field>
                <field var='secret'>
                    <value>SECRET</value>
                </field>
            </x>
        </publish-options>
    </pubsub>
</iq>

[tmolitor-stud-tu]

But: don't send too much of this pushes in a row (for example when XEP-0198 stream management decides to hibernate the stream having multiple messages still being unacked) because every push will create a visible notification on iOS.

This scenario is something you should consider, because TCP needs some time to timeout and all stanzas sent into an already dead but not timed out TCP session must trigger a push, otherwise messages will be seen only when the user opens the app eventually.

Other scenario you should consider to make iOS more "responsive": if a stream management ack is not received for a certain time (in prosody I use 30 seconds as default), send pushes for all messages in the smacks queue (but only once!) and directly trigger pushes for all messages coming after this until an ack is finally received or the session gets hibernated.

If you don't do this, depending on the TCP timeout (up to 8 minutes) the conversation will be disrupted for that long (or even longer if you don't handle pushes for stanzas still in the queue when a session gets hibernated).

[ioscanner]

Here is the trace: 2020.04.16 18:49:38 TRACE [socket_c2s_ssl-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.Push0IQHandler - intercepted

http://jabber.org/protocol/pubsub#publish-options removed

2020.04.16 18:49:38 TRACE [socket_c2s_ssl-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.PushServiceManager - User 'user' has 1 push notification services configured. 2020.04.16 18:49:38 DEBUG [socket_c2s_ssl-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.Push0IQHandler - Push service 'ios13push.monal.im', node '4B179F90-77E4-4EE9-A7B2-removed', for user 'user' was already registered. 2020.04.16 18:49:38 TRACE [socket_c2s_ssl-thread-2]: org.igniterealtime.openfire.plugins.pushnotification.Push0IQHandler - intercepted

2020.04.16 18:51:41 DEBUG [component-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.PushInterceptor - Delivered a notification for user 'user' to node '4B179F90-77E4-4EE9-A7B2-removed' on service 'ios13push.monal.im'. 2020.04.16 18:51:41 TRACE [component-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.PushInterceptor - For user 'user', Routing push notification to 'ios13push.monal.im': http://jabber.org/protocol/pubsub#publish-optionsremoved 2020.04.16 18:51:41 TRACE [component-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.PushInterceptor - For user 'user', found publish options for node '4B179F90-77E4-4EE9-A7B2-removed' of service 'ios13push.monal.im' 2020.04.16 18:51:41 TRACE [component-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.PushInterceptor - For user 'user', found node '4B179F90-77E4-4EE9-A7B2-removed' of service 'ios13push.monal.im' 2020.04.16 18:51:41 DEBUG [component-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.PushInterceptor - Delivered a notification for user 'user' to node '24BF4538-3488-46F5-9800-removed' on service 'ios13push.monal.im'. 2020.04.16 18:51:41 TRACE [component-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.PushInterceptor - For user 'user', Routing push notification to 'ios13push.monal.im': http://jabber.org/protocol/pubsub#publish-optionsremoved 2020.04.16 18:51:41 TRACE [component-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.PushInterceptor - For user 'user', found publish options for node '24BF4538-3488-46F5-9800-removed' of service 'ios13push.monal.im' 2020.04.16 18:51:41 TRACE [component-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.PushInterceptor - For user 'user', found node '24BF4538-3488-46F5-9800-removed' of service 'ios13push.monal.im' 2020.04.16 18:51:41 TRACE [component-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.PushInterceptor - For user 'user', found service 'ios13push.monal.im' 2020.04.16 18:51:41 TRACE [component-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.PushInterceptor - For user 'user', 1 push service(s) are configured. 2020.04.16 18:51:41 TRACE [component-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.PushServiceManager - User 'user' has 1 push notification services configured. 2020.04.16 18:51:41 TRACE [component-thread-3]: org.igniterealtime.openfire.plugins.pushnotification.PushInterceptor - If user 'user' has push services configured, pushes need to be sent for a message that just arrived.

[anurodhp]

your server is sending too many pushes for this token. I searched the logs and i see it hitting the rate limit. The limit max 5 pushes for an account per second and i am seeing way more than that.

Apr 16 05:41:46 ios13push.monal.im:push_appserver warn Rate limit for node '4B179F90-77E4-4EE9-A7B2-' reached, ignoring push request (and returning 'wait' error) Apr 16 05:41:46 ios13push.monal.im:push_appserver warn Rate limit for node '4B179F90-77E4-4EE9-A7B2-' reached, ignoring push request (and returning 'wait' error) Apr 16 05:41:46 ios13push.monal.im:push_appserver warn Rate limit for node '4B179F90-77E4-4EE9-A7B2-' reached, ignoring push request (and returning 'wait' error) Apr 16 05:41:46 ios13push.monal.im:push_appserver warn Rate limit for node '4B179F90-77E4-4EE9-A7B2-' reached, ignoring push request (and returning 'wait' error) Apr 16 05:41:46 ios13push.monal.im:push_appserver warn Rate limit for node '4B179F90-77E4-4EE9-A7B2-' reached, ignoring push request (and returning 'wait' error) Apr 16 05:41:46 ios13push.monal.im:push_appserver warn Rate limit for node '4B179F90-77E4-4EE9-A7B2-' reached, ignoring push request (and returning 'wait' error)

so yes it is hitting monal correctly just too much. Could you try going to notifications under settings and scroll to the bottom and select rebuild. This will give you a new token and allow a fresh start.

[ioscanner]

I am not using it much. It is just me. It seems to send many requests for each message. That was one message. I did try to rebuild the tokens many times. It does the same thing. When I was using prosody I had no issues with push and Monal. It sounds like the push plugin must be doing something. Is there a way to limit how many times it tries per message?

I don't understand why push has been so difficult with openfire. I love the other features it offers, but push notifications not working has been killing me. I have tried everything I can think of. Is there options to tune to improve the stability of pushnotifications?

I appreciate everyones help in sorting this out. I had to write a program that would look for new messages and send me an SMS to tell me I had messages waiting. LOL

[guusdk]

Thanks for all of the feedback guys. I've raised several issues in Openfire's PushNotification plugin issue tracker. I'd appreciate continued feedback in that tracker, where appropriate.

I'm sorry that this is causing you grief, @ioscanner. Openfire's support is lacking, as what's implemented now was created as a best-effort approach in literally one afternoon, without any iOS device to test on. We've done some very basic testing, but since its inception, the discussion in this thread is the first constructive feedback loop that it had.

Improving support in Openfire requires resources. That can come in many shapes:

• Providing constructive feedback, supplying us with test reports, filing issues;
• Submitting patches, pull requests to improve the code;
• Motivating other people to do the above.

To me, this has been an labor of love, meaning that it gets put to the back burner a lot. Paid work obviously takes precedence, but outside that, this feature competes with the gazillion other pet projects and issues deemed important for various reasons. To me, it's a matter of prioritization, and, frankly, given the lack of feedback that I've had up until now, push notifications haven't been high on my priority list.

[tmolitor-stud-tu]

@anurodhp the too much should not harm here (but should be solved on the openfire side of things of course, see my last comment: https://github.com/anurodhp/Monal/issues/354#issuecomment-614562560

The problem is that the pushes are all silent, thus no visible indication of new messages is shown...and: if monal isn't running in the background it will not show any notification at all (I'm not sure if it will show notifications after silent pushes when in background given the ios13 push changes you made).

[ioscanner]

I don't do much development in Java, but I am working on digging thru the code. I am willing to start helping in anyway I can. I have windows, mac, linux, android and IOS to test from. Do you have a chat (discord, xmpp, IRC, slack) setup for development? I would like to help. I do have servers in 3 data centers.

[guusdk]

Do you have a chat (discord, xmpp, IRC, slack) setup for development?

Sure, you can join us in the open_chat@conference.igniterealtime.org chat room. That MUC revolves around all development for Openfire, Smack and related projects.

[Echolon]

@tmolitor-stud-tu @ioscanner @guusdk

Thank you guys for evaluating - is that still an issue on Monal side or has it "moved" to Openfire?

[guusdk]

I think this definitely needs Openfire improvements.

[Echolon]

Okay alright, the I will close. Comeback any time if there are news!