search

element-hq / element-android

A Matrix collaboration client for Android.
https://element.io/
GNU Affero General Public License v3.0
3.39k stars 735 forks source link

UnifiedPush (ntfy) not working #7069

Open MrRulf opened 2 years ago

MrRulf commented 2 years ago

Steps to reproduce

  1. Set up/start Element from f-droid
  2. Set notification method to unifiedPush (ntfy in my case)
  3. Close Element
  4. Ask somebody to write u a message
  5. Check if there is a notification (in my case, none)

My guess is that the problem is on Elements side because I'm only having this issue since the last update (1.4.34)

Outcome

What did you expect?

To get notifications

What happened instead?

I get no notifications

Your phone model

Fairphone 4

Operating system version

Android 12

Application version and app store

Element - Secure Messenger version 1.4.34 from f-droid

Homeserver

matrix.org

Will you send logs?

No

Are you willing to provide a PR?

No

MrRulf commented 2 years ago

Forgot to add, using the background sync of Element still works.

MrRulf commented 2 years ago

If somebody explains me how I can send useful logs I'm totally willing to send them, I just picked No because I think they won't help much, since the notification isn't broken or anything, just delayed until I open the app. About the PR, if I can help in any way, same as with the logs, I just don't think I'd be able to find the error in the code and fix it, why I picked No to give no wrong expectations.

blacklight commented 2 years ago

I can confirm the issue. ntfy push notifications used to work before the latest update, they stopped working after updating, and background sync is still working.

Are these things properly tested before pushing out a new stable release?

p1gp1g commented 2 years ago

176.126.240.158 got banned by ntfy.sh after too many requests, and looks like a matrix.org IP (https://github.com/matrix-org/synapse/issues?q=is%3Aissue+176.126.240.158).

Is there a list of matrix.org IP to whitelist them on ntfy.sh ?

p1gp1g commented 2 years ago

We need the ones for push requests (outgoing connections)

blacklight commented 2 years ago

Hmm this is weird. I use my own ntfy server for push notifications, and that's what my Android app is registered to. Does the IP blacklist on ntfy.sh impact also personal servers?

p1gp1g commented 2 years ago

It is possible: check the doc https://ntfy.sh/docs/config/#rate-limiting

binwiederhier commented 2 years ago

This should be resolved now. I whitelisted the matrix.org IP address using this https://github.com/binwiederhier/ntfy/commit/ed4cc86c5c28633e4a8809e5aa1af8be7bc903f0. Thanks @p1gp1g for the pointers.

p1gp1g commented 2 years ago

If you want to do the same setup than ntfy.sh:

/etc/ntfy/server.yml

visitor-request-limit-exempt-hosts: "176.126.240.158"

/etc/nginx/nginx.conf

        geo $limited {
                default 1;
                176.126.240.158/32 0;
        }
        map $limited $limitkey {
                1 $binary_remote_addr;
                0 "";
        }
        limit_req_zone $limitkey zone=one:10m rate=1r/s;

You should add ipv6 too your server supports it (2a00:1098:84:1c8::158)

Please not these IPs were discovered with the requests from the matrix.org server

huguesdk commented 2 years ago

i have the same problem. i’m using my own ntfy server. could it be a problem related to rate limiting when fluffychat notifications (that use the same ntfy server) continue to work?

i’ve noticed quite slow sync times recently. i don’t know whether it’s a client or server problem, though. i run my own synapse homeserver.

to be sure to get notifications on android, i use element, fluffychat and schildichat at the same time. i usually get 1 or 2 notifications, and not always from the same clients. it’s sadly funny.

people trying to use element instead of whatsapp or signal keep complaining to me that they often don’t receive notifications. would it be possible to have a safety measure that would ensure to at least perform a sync every hour or so, to catch messages for which notifications failed? i know android is quite aggressive regarding battery optimizations, but maybe it’s nevertheless possible?

binwiederhier commented 2 years ago

@huguesdk I recommend enabling debug (or trace mode) in ntfy for a while to see where the issue is. It's really verbose so you should be able to spot a problem if there is one in ntfy. Don't forget to turn it off though, because otherwise your logs will fill up.

kojid0 commented 1 year ago

Any news on this? Element is not an instant messenger anymore... This should get maximum priority.

bahur142 commented 1 year ago

I've never had any problems since I get using ntfy. I self host both - synapse and ntfy. Using schildichat fork instead element. Non of my friends complains about notifications. Are you sure your ntfy app shows silent notification "Listening for incoming notifications"? This should be enabled. Because if this silent notification is suppressed, you could miss some notifications on element. Also you should set ntfy app in android settings "battery optimization" to "Not optimized" to ensure the app will stay awake and not killed by the os.

DeepBlueBubble commented 1 year ago

Mine is also not working. I am using the FCM distributor one. When the App is in the background the notification will never arrive. It can receive when in the foreground. However when I switch to the Nextpush it works nomral.

dessalines commented 1 year ago

Same, I just switched from google's fcm to ntfy. The element subscription shows up in ntfy android, but no notifications come through until I manually open element.

binwiederhier commented 1 year ago

For anyone using the ntfy.sh server and not a selfhosted instance: I am seeing lots and lots of rate limited (aka rejected) requests from various matrix instances. There are very few whitelisted Matrix servers (matrix.org being one of them), so it may very well be the case that you are just a victim of rate limiting on the ntfy.sh server.

This only affects the public ntfy.sh instance. If you are selfhosting ntfy, i suggest you turn on debugging/tracing and look at the logs.

(I am the ntfy maintainer)

dessalines commented 1 year ago

I'm using the matrix.org homeserver, and I don't get notifications until I open up the element app. In the ntfy android app, it says ntfy.sh/[MY_ID_HERE] .... im.vector.app (UnifiedPush), so I assume everything is set up correctly, but this is my first time trying it out.

Is anyone correctly getting ntfy notifs?

binwiederhier commented 1 year ago

I'm using the matrix.org homeserver, and I don't get notifications until I open up the element app. In the ntfy android app, it says im.vector.app (UnifiedPush), so I assume everything is set up correctly, but this is my first time trying it out.

Is anyone correctly getting ntfy notifs?

You can enable logging in ntfy, and see if ntfy receives and forwards the notifications. Go to Settings -> Record logs, and then after you should have received a notification, you can Copy or Upload the logs. That should tell you more.

krair commented 1 year ago

This could be an entirely separate issue, but wanted to add my findings for posterity. I was also having this issue:

Setup:

Annoyingly the Element android app was able to receive the test notification from the "Troubleshooting Notifications" test. But I was not receiving message notifications. Looking through the homeserver.log makes me wonder if this test uses both my ntfy gateway and the matrix.org gateway https://matrix.org/_matrix/push/v1/notify.

I found my log was filled with:

synapse.http.client - 469 - INFO - httppush.process-2 - Error sending request to  POST https://ntfy.mydomain.tld/_matrix/push/v1/notify: DNSLookupError no results for hostname lookup: ntfy.mydomain.tld

For me the solution was to both:

Restarted the synapse container and started receiving instant notifications on the android app.

FrederikSchack commented 1 year ago

I'm using Element and Fluffychat and NTFY with the default server ntfy.sh. It's running on LineageOS on my Poco X3 Pro. It worked great for about a month, until I updated Element, but now I have the same issue, I don't always get notifications and friends are complaining that they don't receive notifications.

It further seems to me that there's no open source application that can get notifications reliably through (also Tox, Jami, XMPP and many others).

I'm not a developer, but I suspect that LineageOS is suffering under the same energy optimizations as Android: https://github.com/vector-im/element-android/blob/develop/docs/notifications.md#how-does-a-matrix-client-get-a-message-from-a-homeserver?

binwiederhier commented 1 year ago

ntfy.sh + Element works for me and many others. Try re-subscribing in Element by re-selecting the Notification method.

Also: I updated the Matrix code in ntfy a little which will make it less likely that notifications stop working. This was rolled out less than 24h ago, so it's worth a shot to retry it.

(FYI: I am the ntfy maintainer)

kojid0 commented 1 year ago

Try re-subscribing in Element by re-selecting the Notification method

Unfortunately this didn't work for me and 2 friends (all Google Pixel phones with Android 13). Ntfy is still "working" in the background and is not being killed by the battery manager

teohhanhui commented 1 year ago

Unfortunately this didn't work for me and 2 friends (all Google Pixel phones with Android 13).

I'm on the Pixel 6a. What worked for me was to run "Troubleshoot Notifications" in Element. It detected a problem in "Endpoint Registration" and gave me a button to re-register that. Notifications have been working fine since then.

casta commented 1 year ago

Hello,

It seems I face the same issue. Using the Fdroid Element build. In my case, I’m using NextPush with personal matrix gateway configured. Messages correctly flows from matrix homeserver (matrix.im) to my push server, and from my push server to my android device.

Just to be sure that my NextPush client has not been shut-down by android, I restart it, then I send a message. Here is the logcat I get:

03-10 12:29:58.716 27157 28746 D RestartWorker: Restarting
03-10 12:29:58.717 27157 28746 D StartServiceCompanion: Starting the Listener
03-10 12:29:58.717 27157 28746 D StartServiceCompanion: Service is started: false
03-10 12:29:58.717 27157 28746 D StartServiceCompanion: nFails: 0
03-10 12:29:58.721 27157 27179 I WM-WorkerWrapper: Worker result SUCCESS for Work [ id=da8b502d-9a38-488e-b9be-b99af2faf3b1, tags={ org.unifiedpush.distributor.nextpush.services.RestartWorker } ]
03-10 12:29:58.727 27157 27157 I StartService: StartService created
03-10 12:29:58.733 27157 27157 D StartService: onStartCommand
03-10 12:29:58.734 27157 27157 D RestartNetworkCallback: Registering new ConnectivityManager
03-10 12:29:58.735  1833  3394 D ConnectivityService: requestNetwork for uid/pid:10327/27157 activeRequest: null callbackRequest: 1292 [NetworkRequest [ REQUEST id=1293, [ Capabilities: INTERNET&NOT_RESTRICTED&TRUSTED&NOT_VCN_MANAGED Uid: 10327 RequestorUid: 10327 RequestorPkg: org.unifiedpush.distributor.nextpush] ]] callback flags: 0 priority: 2147483647
03-10 12:29:58.738 27157 27254 D RestartNetworkCallback: Network is CONNECTED
03-10 12:29:58.738 27157 27254 D RestartNetworkCallback: Network Capabilities changed
03-10 12:29:58.745 27157 27157 D Api     : cSync done.
03-10 12:29:58.974 27157 27176 D OpenGLRenderer: endAllActiveAnimators on 0x77548176c0 (MenuPopupWindow$MenuDropDownListView) with handle 0x76647ee2e0
03-10 12:29:58.983 27157 19737 D FailureHandler: newEvent/Eventsource: okhttp3.internal.sse.RealEventSource@563fe3f
03-10 12:29:58.987 27157 19737 D SSEListener: onOpen: 200
03-10 12:29:58.987 27157 19737 D SSEListener: New SSE message event: start
03-10 12:29:58.992 27157 19737 D SSEListener: New SSE message event: keepalive
03-10 12:29:58.996 27157 19737 D SSEListener: New keepalive: 300
03-10 12:30:33.014 27157 19737 D SSEListener: New SSE message event: message
03-10 12:30:33.030 27157 19737 D Distributor: Message forwarded

Here is the corresponding message flowing through my NextPush server:

2a03:4000:60:d87:a494:8ff:fe4d:99b6 - - [10/Mar/2023:12:30:33 +0100] "POST /_matrix/push/v1/notify HTTP/1.1" 200 35 "-" "Synapse/1.77.0"

You can see the socket is restarting is setting up with initial messages (correctly setting keeplive to 300), then the 2 last lines, the NextPush client receive a message (from my matrix home server through my personal NextPush server)

The last logcat line is the forward of the message to the application. Then… nothing happens. No reaction from Element. But:

If I uses the Playstore version with FCM push (I have microg on my phone), there are no such issues. So I would suspect something around the UnifiedPush listener activity that prevent Element to wake up ?

p1gp1g commented 1 year ago

If I swipe Element from the tasks, notifications still appears for some time, but at some point, do not appears any more. Probably after Android started dozing Element.

Can you check https://dontkillmyapp.com/ ?

This looks like the behavior of some vendor's aggressive battery "optimization". Some of the vendors even set applications in user force-stopped state which prevent other apps from starting it. Preventing user force-stopped application from starting is a good thing when the user has force stopped it :facepalm:

casta commented 1 year ago

I should not be affected by this since I’m running LineageOS 19 with microg, and I have disabled battery optimization for Element (which should not be necessary).

And remember: I do not have this behavior with the PlayStore build with FCM. Element is perfectly woken-up by the event from FCM (but I would like to avoid using FCM…)

p1gp1g commented 1 year ago

MicroG is a system app and is allowed to wake up app even in force-stopped state.

It would be interesting if you could get logcat of nextpush and element + nextcloud log when you don't receive notifications. Which version of nextpush/element do you use ?

Edit: As @binwiederhier said: we should move the discussion related to NextPush to https://matrix.to/#/#unifiedpush:matrix.org

binwiederhier commented 1 year ago

It feels like this issue has evolved and is no longer about ntfy not working, right? I would suggest to move troubleshooting of NextPush to the UP matrix room, and troubleshooting of ntfy to the ntfy Discord.

(I am the ntfy maintainer, and I'm happy to help with ntfy issues. But this is the Element GitHub issue tracker, so I feel like this discussion is misplaced. I'm going to unfollow the issue.)

kojid0 commented 1 year ago

I'm on the Pixel 6a. What worked for me was to run "Troubleshoot Notifications" in Element. It detected a problem in "Endpoint Registration" and gave me a button to re-register that. Notifications have been working fine since then.

Unfortunately this didn't help. I still don't get any notifications in Element (F-Droid build). I'm on CalyxOS with MicroG, running Android 13. Ntfy standard server, both Element and ntfy are excluded from battery optimizations. Endpoint registration seems to be fine (Element notification troubleshooting).

casta commented 1 year ago

Not sure it’s specific to NextPush. I think I have the same behaviour with ntfy and UP-FCM (https://unifiedpush.org/users/distributors/fcm/).

After digging a bit more, it seems that the notification is broken as soon as Element is not running. For example just after booting the phone, on after killing the Element process. Then I have a strange behavior: “sometime”, the message is lost, and “sometime”, the broadcast intent is caught and element is starting, and the notification appear, with this logs:

03-10 15:24:49.179  4447  4930 D Distributor: Message forwarded
03-10 15:24:49.180  1849  1943 D CompatibilityChangeReporter: Compat change id reported: 143937733; UID 10326; state: ENABLED
03-10 15:24:49.200  1849  1943 I ActivityManager: Start proc 6180:im.vector.app/u0a326 for broadcast {im.vector.app/im.vector.app.core.pushers.VectorUnifiedPushMessagingReceiver}

For me it could be more like Element not intercepting the UnifiedPush broadcast intent under some conditions, or something like this.

LukeWheeldon commented 1 year ago

Same here. Using grapheneOS. Any updates?

Other ntfy notifications are going through as I can randomly test anytime curl -d "Hi" ntfy.sh/bw170randomJfJlx and it will pop-up on the phone, but I'm not getting the element ones.

MelBourbon commented 1 year ago

Since a few days ntfy method is also not working anymore with Android 13 (graphene OS). Google Service also not working. So currently the only way to get reliable notifications for Element is using background sync. Other users of my server have the same issue.

dessalines commented 1 year ago

I tried this again, and can verify its still broken. @casta is correct that:

After digging a bit more, it seems that the notification is broken as soon as Element is not running.

While element is in the foreground / not closed, I get ntfy notifs. As soon as its killed in the background, the notifs stop.

If the ntfy is wired to "wake up" element when it receives a notif, its not doing so.

ismay commented 1 year ago

I've been encountering this problem as well. Strange enough after some fiddling around it's working now. I'm not sure why it's working now, but I get the idea that it wasn't working because I also had other element instances signed in to the same server on other devices.

It's a long shot, and I'll keep an eye on it, but it seemed like notifications for element on my phone with ntfy (without element running in the background) started working again when I closed everything else that was signed into the same account on other devices. Also, it doesn't seem to be ntfy specific, because I was observing the same failure to notify on mobile when element was connecting directly to the server via polling. So it seems to be an element bug, not ntfy.

FrederikSchack commented 1 year ago

I still have issues with notifications and calling, running on Lineage OS for MicroG with NTFY. My first experience with Lineage OS was good in January and a bit into february, calling could be slow, but it worked. Since then I could rarely get a call through and messages can be very slow at arriving, like several minutes. Sometimes I see a notification about a message and I can see the first bit of the message, but I can't see the message inside Element and it can take yet more minutes before the message appears inside Element.

I also have Element installed on a Windows PC.

FrederikSchack commented 1 year ago

"Matrix largest home servers are incredibly slow to use": https://www.youtube.com/watch?v=aVwl892hqb4

kinslayer1982 commented 11 months ago

Noticed the same problem with Element and ntfy.

I'm running a matrix server (synapse), ntfy server and fmd server behind a nginx reverse proxy. Element, ntfy and fmd from F-Droid on a Galaxy S21 FE (Android 13). After setting up ntfy I got notifications from Element for a short time, then they stopped working while for fmd they were still working. Now that I've re-registered ntfy with Element again, they started working again. I will observe if they keep working or stop again.

Edit: 4 days now and still seems to be working. Don't know why notifications in Element stopped working while others didn't.

cal-andrew commented 8 months ago

It started working for me when I disabled battery optimisations for Element, as well as ntfy

I am using the F-Droid version of Element on CalyxOS with microg. The Google Play version of Element could also get push notifications through microg, without disabling battery optimisations.

microg, being a system app, may have permissions to wake up apps with battery optimisations, while ntfy and other user-installed apps don't. If so, custom ROMs could fix this by including ntfy (or some other UP distributor) by default.

luk6173 commented 7 months ago

Have same issue with ntfy. Running Element F-Droid Version (latest), also latest NTFY on my Android Smartphone and on my Server.

Is there any fix planned next time ?

p1gp1g commented 7 months ago

@luk6173 if you run your own synapse server, you have to whitelist your ntfy ip address: https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#ip_range_whitelist

luk6173 commented 7 months ago

@luk6173 if you run your own synapse server, you have to whitelist your ntfy ip address: https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#ip_range_whitelist

@p1gp1g adding my NTFY IP from my Server or from my Smartphone (Android App) ?

mpeter50 commented 6 months ago

It seems this issue has reports about a lot of issues for which the cause cant be determined. Matrix notifications through Ntfy are not working for me either, but I'm sure that the Ntfy app itself is working fine, below I'll tell you why.

I think I have found a few ways to test whether a few very specific parts of the notification pipeline are working or not. An important tool in these tests is the Ntfy webapp: https://ntfy.sh/app This is hosted by binwiederhier (lets hope that if I dont use an @ sign you wont get unnecessarily mentioned), who is the developer of Ntfy, and who runs the main Ntfy instance that is used by the matrix.org homeserver, so this webapp is safe to use.

I have written the below guide

It got to be a bit long, so instead of dumping it here you can read it in this gist: https://gist.github.com/mpeter50/9220dac1056c9a66c313d049838c1ab2

scottwallacesh commented 6 months ago

I think this is "simply" a problem of Element/Schildichat not waking when ntfy receives the push notification.

If you disable battery optimisation on Element/SC and allow it to run in the background, everything works as you would expect.

mpeter50 commented 6 months ago

I have it disabled for SC already. Also, I didn't receive the notification in the Ntfy webapp either, when I was watching the topic of the push key while sending messages to my account from an other one.

p1gp1g commented 6 months ago

@luk6173 if you run your own synapse server, you have to whitelist your ntfy ip address: https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#ip_range_whitelist

@p1gp1g adding my NTFY IP from my Server or from my Smartphone (Android App) ?

This should be your ntfy IP as seen by your synapse server. If you look at your synapse logs

p1gp1g commented 6 months ago

For helping many users, especially for users self-hosting their services, the problem is most of the time a not working configuration.

To troubleshoot UnifiedPush for self-hosters:

First, __if you self-host your homeserver and your push server: there is a high probability you have not set the ip_range_whitelist__. Check your synapse's logs.

Then, you should run the notifications troubleshooter feature on element.

If you don't receive the notification, the problem comes from your push provider:

If you receive the test notification:

* Most common issues

luk6173 commented 6 months ago

This should be your ntfy IP as seen by your synapse server. If you look at your synapse logs

I assume then I have to whitelist 0.0.0.0 because my ntfy IP from my Android Smartphone App changes often, but should not be a problem because of user based access

p1gp1g commented 6 months ago

Your ntfy server IP as seen by your synapse server. It may be a private IP depending on your setup. You should check the logs

luk6173 commented 6 months ago

Your ntfy server IP as seen by your synapse server. It may be a private IP depending on your setup. You should check the logs

thank you for your clarification, I go now and setup it

luk6173 commented 6 months ago

For helping many users, especially for users self-hosting their services, the problem is most of the time a not working configuration.

Thank you very much this helped me to get it running and to understand how it works!