Trouble with Push Notifications when Advanced DNS filtering is ON

Artim96 commented 8 months ago

Hey, maybe someone has experience with this because I'm at a loss. For a few weeks now I often have problems receiving push notifications from pretty much any app (Telegram, WhatsApp, Discord, PayPal, GMail, you name it). Regularly, they only appear when I actively open the app. But I can't figure out, what exactly is causing it. In DNS log there aren't that many blocked domains (after I cleaned up some block lists), the only Google related ones I see are googletagmanager, adservice.google.com and so on.

For all I know, any Android App that wants to have reliable push notifications is being urged to send them via Googles GCM service for years now. So my theory is that something is interfering with that service, but I can't tell what. I've already tried using Rethink DNS Plus, only enabling block lists in its settings and disabling the on-device blocklists, as the app says this can improve firewall rules, but no change.

Does anybody know what domain or (system) app I could whitelist to see if that helps? I can't tell for sure that the app the the cause, but I'm at least 95 % sure. I have attached a bug report, maybe someone sees something.

rethinkdns.bugreport.zip

ignoramous commented 8 months ago

We had a few reports about push notifications not coming through, but these remain a mystery to us as well, as we don't see such behaviour on any of our devices. We'll keep looking.

Btw, did this behaviour only start happening with v055a?

Have you changed any setting in Configure -> Network from its default?

Artim96 commented 8 months ago

I'm at least 90% sure it did, yes. It's not impossible it started with v055, but I don't really think so

ignoramous commented 8 months ago

Okay, I am going to park this one as v055b is super close, which has undergone major changes in the network engine. Hopefully, by magic, v055b has fixed it; if not, I'll have to start looking for answers, as this issue sounds worrisome.

Artim96 commented 8 months ago

Sounds like a plan.

Btw if it helps, I'm on a Pixel 6 and have been on the Android 14 Betas and am now on the Feature Drop Betas. Not sure what the other affected people use, but if they are on 14 too but your devices aren't, that might be the reason that you aren't affected.

Mygod commented 8 months ago

v055a and Android 14 here and can confirm the same issue. I'm using system/Cloudflare DNS without block lists and this still happens, so it seems like the issue is caused by the app itself.

Mygod commented 8 months ago

I am starting to think that this is a Google issue. Seeing the same issue using a different VPN app.

privat3dancer commented 8 months ago

Trouble with notifications started for me with the first Android 14 QPR1 Beta update. Previous Beta versions were fine. It doesn't seem to be solely an issue of notifications. For instance, gmail is actually not syncing reliably except when manually refreshed while the VPN is running.

ignoramous commented 8 months ago

I know that UDP sendmmmsg and recvmmmsg syscalls have trouble working with Rethink. Important that it works because of HTTP3 which uses UDP. I'm unsure if that is the cause for what we are seeing with notifications, because apps almost always fallback to HTTP2 or WebSockets (in case of long open connections).

If you're technical enough, can you adb logcat and see what errors/warnings prop up for Gmail?

Artim96 commented 8 months ago

I just recorded a log for GMail. I recorded through Shizuku on device, I had the log include Package Names, Tags and Content, If needed, UID, PID and TID can also be included, but also Date and Time.

Here I sent a mail to my GMail adress, which wasn't pushed to my device, so I manually opened the GMail app and the mail was shown immediately. Hope this helps.

GMail_Recording.zip

privat3dancer commented 8 months ago

Yes I'll grab a logcat, but first I want to note that Messages does not have any trouble notifying for SMS/MMS as they arrive. I don't know much of anything about Android internals, but this bug doesn't seem to affect push notifications for all apps. So far I see that apps such as Gmail, Signal, and Duo Push need to be in the foreground to receive new messages and issue notifications.

Mygod commented 8 months ago

I am starting to think that this is a Google issue. Seeing the same issue using a different VPN app.

I take this back. I was trying WireGuard just now and it seemed fine. 🤔

Mygod commented 8 months ago

One thing I noticed is that if I enable advanced DNS filtering, then the FCM connection doesn't seem to work:

tcp6       0      1 ::ffff:192.168.0.:37892 ::ffff:100.71.207.:5228 SYN_SENT    16214/com.google.android.gms.persistent

After turning it off, it seems to work.

tcp6       0      0 ::ffff:192.168.0.:36210 ::ffff:172.253.63.:5228 ESTABLISHED 16214/com.google.android.gms.persistent

privat3dancer commented 8 months ago

Oh @Mygod, I tried turning this off, ~~but it doesn't seem to have any macroscopic effect on this issue for me.~~

Edit: OK I take it back. After stopping and starting the service, I think this may have had a positive effect. I'm seeing more notifications when they should arrive. Note: one thing that is not made clear to the user is which settings take effect immediately and which require restarting the service.

Artim96 commented 8 months ago

One thing I noticed is that if I enable advanced DNS filtering, then the FCM connection doesn't seem to work:
tcp6       0      1 ::ffff:192.168.0.:37892 ::ffff:100.71.207.:5228 SYN_SENT    16214/com.google.android.gms.persistent
After turning it off, it seems to work.
tcp6       0      0 ::ffff:192.168.0.:36210 ::ffff:172.253.63.:5228 ESTABLISHED 16214/com.google.android.gms.persistent
Related: https://firebase.google.com/docs/cloud-messaging/concept-options#messaging-ports-and-your-firewall

This might have been the solution (at least for me). I'll probably have to monitor this a bit more, but right now it seems like everything is working as expected again.

stilsch commented 7 months ago

can confirm what @Artim96 said - I've disabled advanced DNS filtering and the push messages came in promptly.

ignoramous commented 7 months ago

I can confirm that this issue isn't present on v055b, the current dev version, due to be released in a week or two. Hopefully, whatever it is about Advanced DNS filtering that broke push notifications has fixed itself through the many changes the network engine has gone through.

Artim96 commented 7 months ago

I can confirm that this issue isn't present on v055b, the current dev version, due to be released in a week or two. Hopefully, whatever it is about Advanced DNS filtering that broke push notifications has fixed itself through the many changes the network engine has gone through.

Good to know, we'll test in a week or two.

AykutCevik commented 7 months ago

I've disabled advanced DNS filtering and the push messages are working properly. I'm on v055a

luckygitt commented 6 months ago

OK, I had the same "Push" problem and tried everything to trace the culprit. Finally, I discovered that blocking the domain "mtalk.google.com" was the culprit. Unblock it and all should be fine. HTH Lucky

ignoramous commented 3 months ago

v055c fixes issues with Advanced DNS filtering which caused push notifications to break.
Some have reported that turning OFF Prevent DNS leaks fixes push notifications for certain apps like Signal and Proton (might be (ab)using port 53 for maximum network jujitsu).
Some users inadvertently blocked domains that blocked notifications. Solving for this needs correction on per-user basis (although, Rethink could do better and alert users when they block critical domains or IPs: #54).
Some have reported that Block when DNS is bypasssed Universal (global) firewall rule also causes lost notifications in some apps.

Please feel free to re-open if notifications are still kaput.

Artim96 commented 3 months ago

It seems I have to reopen this issue. After your comment on fixing it in v055c I reactivated Advanced DNS filtering and it causes the exact same issues. Deactivating it fixed it immediately. Logs are attached, although due to only short testing I can only confirm for sure that this does interfere with Telegram. rethinkdns.bugreport.zip

EDIT: also causing the issue with GMail.

Artim96 commented 3 months ago

It seems even turning off advanced DNS filtering doesn't help with GMail this time. Other apps seem to be unaffected though. rethinkdns.bugreport.zip

satuday commented 1 month ago

I recently switch from moto edge 2022 to pixel 8 and notice notification are not coming in. Seems like my new pixel 8 with android 14 is effected, but my previous moto edge 2022 with android 13 did not have this issue.

ignoramous commented 1 month ago

Worrying.

Notification for which app? All apps?
Can you confirm if you haven't blocked Google Play service with any Firewall or DNS (domain) rules?
Make sure Prevent DNS leaks and Advanced DNS filtering in Configure -> DNS is turned OFF?
And that you're not blocking some critical domains by using aggressive blocklists with RDNS+ or On-device blocklists.

satuday commented 1 month ago

Apps that I noticed notifications not showing are Google voice, Gmail, okta, Amazon, wyze, but I'm pretty sure there's more I didn't noticed.

I didn't setup any additional firewall or DNS rules other than what is on default.

I turned off prevent DNS leaks and advanced DNS filtering and I tested with Google voice and okta and Gmail and I'm getting notifications now for those now.

No, I only have the default DNS and firewalls on.

ignoramous commented 1 month ago

Apps that I noticed notifications not showing are Google voice, Gmail, okta, Amazon, wyze, but I'm pretty sure there's more I didn't noticed.

No notifications with Advanced DNS filtering and Prevent DNS leaks ON, correct?

satuday commented 1 month ago

yup.

celzero / rethink-app

Trouble with Push Notifications when Advanced DNS filtering is ON #1115