Apple Services after not available after some time. Only renewing the internet connection will solve it.

pictosun commented 6 months ago

Platform

macOS

Protocol

DNS-over-HTTPS

Do you use AdGuard app?

No I don't

Your configuration

macOS/iOS (newest system update)
using .mobileconfig Profiles on macOS/iOS (and also tried AdGuard Pro App with AdGuard DNS as native profile)
tried DoT/DoH with .mobileconfig and DoQ (with a 3rd party app and without .mobileconfig)
block Apple private relay function is 'not' used (and I do have apple services on my whitelist)

Traceroute to AdGuard DNS

./. doesn't matter

Issue Details

While using iOS/macOS devices after several amount of time (especially when changing networks from mobile to WIFI and so on) it does happen that Apple services like App Store, Apple Music do not work anymore.
Even force quitting the apps don't bring back connection to the services
when putting the devices into flight mode and back online the services do work again

Expected Behavior

It should work all the time

Actual Behavior

When using AdGuard DNS (with some blocklists) the problem exists and reappears often.

Screenshots

No response

Additional Information

When using some other DNS services like NextDNS, ControlD etc. (nearly same blocklists enabled) the issue is not existent

It looks like it's something correlated to AdGuard DNS service at all?

Chinaski1 commented 6 months ago

Hello there!

Please clarify the following information:

Do you use third-party software like a VPN?
Which blocklists are enabled?
What DNS server are you connected to?

pictosun commented 6 months ago

Do you use third-party software like a VPN?

No 3rd party tools

Which blocklists are enabled?

HaGeZi Pro & TIF, DynDNS, DNS/VPN/TOR/Proxy Bypass and Gamling | Smart-TV Blocklist for AdGuard Home (by Dandelion Sprout)

What DNS server are you connected to?

dns2-dp-fra-5

Chinaski1 commented 6 months ago

To start, the easiest test is to try to disable all third-party filters except AdGuard DNS and check if the issue persists. In turn, we'll investigate the situation on our side.

pictosun commented 6 months ago

Will try to look at it with iOS 17.5 and macOS 14.5 installed. Maybe it's a Apple bug which has been resolved?!

pictosun commented 6 months ago

@Chinaski1 I can give you the update, that the issue just appeared again when changing WAN networks.

I also checked the query logs and cannot see any block within this time and/or domains... So the blocklists shouldn't be the issue and it looks like it is something else.

Changing DNS profile to some other provider solves the issue. Going back to AdGuard DNS after that also works.

EDIT: Found out, that those issues also have on my Teamviewer App (logged out and said no service possible) after changing the WAN network. So definitively not an Apple only thing.

pictosun commented 6 months ago

Just do give you an update over here... When changing to an other DNS service (Contr...D, Ne...DNS) I do not have those issues (using nearly the same blocklists) and I think it's not related to blocklists itself, as there is nothing showing up within blocked queries.

So something must be different within AdGuard DNS? Could it maybe something correlated to this: https://github.com/AdguardTeam/AdGuardDNS/issues/227

Chinaski1 commented 6 months ago

At this stage it is difficult to say what may be causing the issue. It is not reproduced on our side, we are looking into it.

hagezi commented 6 months ago

I use AdGuard DNS via mobileconfig on 10 Apple devices, including iPhones, iPads, Macbooks and Apple TV and do not have this problem and cannot reproduce it.

pictosun commented 6 months ago

Thanks for your feedback. Really a strange behavior. For me I can more or less reproduce it on several devices with several kind of blocklists (don’t think the lists are the issue).

What I (maybe) found out is, that it is eventually related to changing WAN networks with IPv4/IPv6 changing. At home I do have dual stack, at work IPv4 and mobile it depends on.

Don’t know if this is maybe the case and has something to do with my issues.

As it happended to several macOS/iOS versions I don’t think that this is the case. Especially while hagezi says that everything is fine for his devices.

Don’t know how can look into more details to report it over here to find the reason why this is happening (and also only happening when using AdGuard DNS - with NextDNS and ControlD and other DNS services I don’t have those issues?!).

As I said, I will try to narrow it down so that finding the reason is easier?!

Thanks for your product and help over here.

Batman2814 commented 6 months ago

Try unblocking this domain "mesu.apple.com" This domain must be unblocked for the Apple TV App to work on Apple devices, but it might require other Apple services.

pictosun commented 6 months ago

Try unblocking this domain "mesu.apple.com" mesu.apple.com is always unblocked

As mentioned above I don't think it's an issue with any blocking list at all, as it does happen regardless of entries in any list. So it must be something else. I think it has to do with some kind of network change (IPv4/IPv6) and maybe AdGuards way (the other players like NextDNS and ControlD,... don't do this) to check other resolvers (Google, Cloudflare) when getting a SERVFAIL response.

Or they're still doing something different within their implementation.

When trying out AdGuard Home on a VPS it also doesn't happen.

And as mentioned above it's quite hard to find out what's causing the issue.

ghost commented 6 months ago

I've been experiencing this for the last few weeks as well when accessing the Apple App Store, it will refuse to connect intermittently.

Many other Apple services seem to be playing up as well.

hagezi commented 6 months ago

When trying out AdGuard Home on a VPS it also doesn't happen.

Are you also using AdGuard DNS (unfiltered) as upstream, if not you should test whether the problem also occurs when using it. If the problem occurs, have a look at the log and check whether one of the corresponding domains returns SERVFAIL as a result.

https://unfiltered.adguard-dns.com/dns-query

pictosun commented 6 months ago

Are you also using AdGuard DNS (unfiltered) as upstream, if not you should test whether the problem also occurs when using it.

Thanks for your feedback. I will give it a try. But what about Bootstrap DNS and Private reverse DNS Servers? At the moment they're running with my Unbound resolver on the VPS. Do I also need to change them to test it?

hagezi commented 6 months ago

No, only the upstream DNS.

pictosun commented 6 months ago

I've been experiencing this for the last few weeks as well when accessing the Apple App Store, it will refuse to connect intermittently.

In which scenarios for you? When changing WAN networks (home WIFI to some other networks), or don't you know?

ghost commented 6 months ago

I've been experiencing this for the last few weeks as well when accessing the Apple App Store, it will refuse to connect intermittently.

In which scenarios for you? When changing WAN networks (home WIFI to some other networks), or don't you know?

Both wifi and mobile, with different providers.

DNS is setup with a configuration profile, so I can rule out AdGuard for iOS, or any other DNS app.

pictosun commented 6 months ago

Are you also using AdGuard DNS (unfiltered) as upstream, if not you should test whether the problem also occurs when using it.

The problem does not occur when using it this way. But I'll keep testing it.

hagezi commented 6 months ago

I assume that you also use your AdGuard Home VPS via mobileconfig to at least use a similar test scenario?

pictosun commented 6 months ago

Yes - same devices. Made a mobileconfig (same protocol DoH) and same blocking-lists (but I don't think it's because of the blocking lists as mentioned above).

pictosun commented 6 months ago

Just to give a short update. Changed back (after no issues when using AGH on VPS with AdGuard Servers as Upstream) to AdGuard DNS.io mobileconfig and the issue came back immediately.

So the issue is definitively somehow within adguard-dns.io implementation.

hagezi commented 6 months ago

@pictosun Are you using a mobileconfig in which the home network WLAN is excluded, i.e. your local DNS is used when you are in the home network?

You could test whether the problem also occurs with a self-generated mobileconfig, you can use the following generator: https://dns.notjakob.com/tool.html

I would test an unsigned mobileconfig.

pictosun commented 6 months ago

Are you using a mobileconfig in which the home network WLAN is excluded, i.e. your local DNS is used when you are in the home network?

No - I didn't exclude it. Scenario at home:

Router: DoT profile adguard-dns.io (because of Router only supporting DoT)
Devices: DoH mobileconfig profile (with exclusions)

At work:

Router: no profile at all
Devices: using same profiles as above

It is happening at work and at home when changing networks. So I don't think it's because of network exclusions or not (and other providers do work - no matter of exclusions or not)

Or did I understand you wrong?

hagezi commented 6 months ago

Strange, try a self-generated mobielconfig, just to rule out that this is the cause. For whatever reason.

What steps are you taking to reproduce the problem?

pictosun commented 6 months ago

What steps are you taking to reproduce the problem?

Normal behavior during the day. Leaving the house (and then checking Apples App Store for connection). Coming back home and so on. Today morning I changed mobileconfig back to adguard-dns.io and after leaving the house and coming back the store was not available (no internet connection). When using AGH via VPS and mobileconfig those issues are not there (same with NextDNS, ControlD and others). Only adguard-dns.io with the issue.

hagezi commented 6 months ago

Crazy, I can't reproduce the problem. I use a generated mobileconfig from AdGuard DNS where only the fritz.box domain is excluded.

Does the problem also occur when you switch from Home to Mobile and back to Home (WLAN off/on), or only when you switch from Home to Mobile to Work and then back to Mobile/Home?

pictosun commented 6 months ago

Does the problem also occur when you switch from Home to Mobile and back to Home (WLAN off/on)

Yes. Most of the time I'm in home office so I can reproduce it quite often.

I also do use the generated mobileconfig from AdGuard. fritz.box domain is not excluded in my case. But don't know if this is the issue?!

pictosun commented 6 months ago

@hagezi

only the fritz.box domain is excluded.

How did you exclude it? Via dashboard > Access settings > Disallowed domains ? Or in any other way?

hagezi commented 6 months ago

I don't think that's causing your problem.

When you create a mobilconfig, there is a link to a profile constructor. You can exclude e.g. local domains so that the local queries are not made via AdGuard DNS, but directly via the assigned default DNS. If fritz.box is excluded, *.fritz.box is resolved via the FritzBox, if the clients have been assigned the FritzBox as DNS server via DHCP.

grafik

pictosun commented 6 months ago

I don't think that's causing your problem.

Thanks. Will test it and give feedback. I also don't think that this is causing the issues.

pictosun commented 5 months ago

@hagezi After some testing I can say, that excluding fritz.box doesn't solve the issue.

Today I also recognized, that it also appears when coming out of flight mode). Maybe because disabling flight mode causes a short connection to LTE/5G and after that jumps to WIFI.

So to reproduce it:

toggle flight mode (getting mobile network and wifi)
go to iOS App Store > no connection
disable wifi > App Store does work
enable wifi again > App Store still works

no blocks within query log or somewhere else.

Really annoying issue!

hagezi commented 5 months ago

@pictosun I have carried out the steps to reproduce the problem several times and cannot reproduce it.

pictosun commented 5 months ago

That really is a strange thing. Don't know what else I can do to troubleshoot and/or to find out, what's going on there?

ghost commented 5 months ago

@hagezi

After some testing I can say, that excluding fritz.box doesn't solve the issue.

Today I also recognized, that it also appears when coming out of flight mode). Maybe because disabling flight mode causes a short connection to LTE/5G and after that jumps to WIFI.

So to reproduce it:

toggle flight mode (getting mobile network and wifi)

go to iOS App Store > no connection

disable wifi > App Store does work

enable wifi again > App Store still works

no blocks within query log or somewhere else.

Really annoying issue!

These steps reproduce this issue for me also.

pictosun commented 5 months ago

Thanks for your feedback. So I'm not alone. Don't know what to do now.

Maybe the developers will give a hint how to check something else or maybe there is somewhere a bug from AdGuard DNS itself?!

Also found out, that there is no update over here since a longer time concerning the code...

pictosun commented 5 months ago

Today I found out, that the issue also exists, when using my WIFI-only iPad within my local WIFI network (and not changing the network). Just had it lying around and then I searched for updates within the iPad OS App Store and it says no connection.

I'm using the same Server (profile) within AdGuard DNS settings. So I checked again the logs and found, that I had a block from 'ca.iadsdk.apple.com' from a list (Hagezi) several time ago.

During the 'outage' of the App Store service (Apple Music was working), I tried it several times (and did not disable/enable WIFI > as this would solve the issue). I checked the settings within the Server (profile) and changed my TTL from 3600 to zero. And after this change the iPad OS App Store started directly.

I checked again the logs and got one more block this time. It was quite close to the other one but little bit different 'tr.iadsdk.apple.com'. But even when blocking this request it works.

So maybe it must be something with setting up TTL and having requests to some domain and during the TTL blocking time apple changing the subdomain of the already blocked domain? Scenario:

App Store says no Internet connection
logs say: ca.iadsdk.apple.com > blocked (a few minutes ago)
changing TTL from 3600 to 0
App Store works
logs say: tr.iadsdk.apple.com > blocked
but system still working

@hagezi Hope this is now a better scenario to find out what's going on there?! I setup a TTL of 3600 for my mobile devices AND also a TTL of 3600 for my router (fritz.box DoT profile). Router and devices nearly have the same lists.

And SORRY for saying that it doesn't show up within my blocking lists (but I never thought, that those iadsdk... domains can have an issue and I also didn't see them because of the TTL and having them little bit more down within my query lists.

@emeritaacuity0u can you check, if you also have setup a TTL for your servers/profiles?
And maybe you can troubleshoot the way I did?

hagezi commented 5 months ago

I also use a block TTL of 3600, I cannot reproduce the problems with it. I can't imagine that this is the cause either.

pictosun commented 5 months ago

Ok - you're using the TTL in both variants? For your router and your clients? Or only within one Server (profile)?

ghost commented 5 months ago

This is my setup. There's no other modifications.

pictosun commented 5 months ago

@emeritaacuity0u Did you try to set TTL to 0?

pictosun commented 5 months ago

As @hagezi said, the TTL was not the point. The issues are still there and happening again and again.

So as my iPad only has WIFI access I can say:

changing TTL doesn't help
blocklist also seems not to be the point
mobile network vs. WIFI jumping through WANs is also not the point (iPad is WIFI only)

I'm a bit loss what it could be else if not some kind of implementation from AdGuard DNS itself (doesn't happen with other providers as already mentioned)

Maybe someone else has an idea how to troubleshoot this issue?

hagezi commented 5 months ago

I can't think of anything else at the moment. If you have blocked Private Relay, you could try to see if the problem still occurs when PR is not blocked.

pictosun commented 5 months ago

No - private relay is not blocked and I do have additional whitelisting of those domains.

I unticked the block private relay thing within AdGuardDNS and these are the additional whitelist rules:

doh.dns.apple.com
mask.icloud.com
mask-api.icloud.com
mask-h2.icloud.com
doh.dns.apple.com.v.aaplimg.com

Today it already happened two times with my MacBook which is within my home WIFI only.

But it looks like it is more and more Apple App Store only. Sometimes it also happens with Apple Music (very rare). But must of the time it is the App Store on all devices.

hagezi commented 5 months ago

I'm at the end of my tether too. I absolutely can't reproduce it, no-one in my immediate and extended family can. On none of the Apple devices and these are some that use AdGuard DNS via mobileconfig.

Have you looked at the result of the requested domains in the log when this happens? Does an IP come back, NXDOMAIN, SERVFAIL or do the requests perhaps not end up at the DNS at all?

pictosun commented 5 months ago

As we can see that also @emeritaacuity0u has those issues I'm not alone at all.

Maybe some developers like @Chinaski1 can jump in and help? Are you working on AdGuard DNS and maybe have looked into those issues? Will we see some changes coming with a new release in future?

@hagezi Concerning '*/cf/tr.iadsdk.apple.com' I do get a blocked result with NOERROR as answers for A,AAAA,HTTPS (it's always 3 requests for apple devices) For 'mask.icloud.com' I do get unblocked and also NOERROR as answers.

I also installed LitteSnitch and tried it via DoQ profile on macOS (LS does use DNS Proxy network extension) and it happens even there. So it's not the DoH mobileconfig alone...

hagezi commented 5 months ago

Test the following on the iPad: Do not use the Fritzbox as the system DNS, but Cloudflare 1.1.1.1, for example, and continue to use the AdGuard DNS mobileconfig as the encrypted DNS.

See if this also occurs with a public DNS, which is then used as a bootstrap for the steering.

Apart from that, I can't really think of anything else you could try.

pictosun commented 5 months ago

Test the following on the iPad: Do not use the Fritzbox as the system DNS, but Cloudflare 1.1.1.1, for example, and continue to use the AdGuard DNS mobileconfig as the encrypted DNS.

I don't get exactly what you're talking about... If I setup a mobileconfig it does use the DNS over the mobileconfig.

You mean leaving the mobileconfig enabled and change WIFI network DNS config from automatic to 1.1.1.1 for example? Or what do you mean?

hagezi commented 5 months ago

Yes, exactly that. The "WiFi DNS" is used as a bootstrap DNS to resolve the DNS domain from the mobileconfig.

pictosun commented 5 months ago

@hagezi It looks like the test does work. When setting for example 1.1.1.1 as DNS within network settings the issue is gone (at least so far).

So now it's getting interesting why this does happen and what's the cause?

hagezi commented 5 months ago

Next, use 1.1.1.1 as upstream in the FritzBox and the FritzBox as ‘WiFi DNS’ as before.

AdguardTeam / AdGuardDNS