search

AdguardTeam / AdGuardDNS

Public DNS resolver that protects you from ad trackers
https://adguard-dns.io/
GNU Affero General Public License v3.0
787 stars 62 forks source link

Apple Services after not available after some time. Only renewing the internet connection will solve it. #777

Open pictosun opened 6 months ago

pictosun commented 6 months ago

Platform

macOS

Protocol

DNS-over-HTTPS

Do you use AdGuard app?

No I don't

Your configuration

Traceroute to AdGuard DNS

./. doesn't matter

Issue Details

Expected Behavior

It should work all the time

Actual Behavior

When using AdGuard DNS (with some blocklists) the problem exists and reappears often.

Screenshots

No response

Additional Information

When using some other DNS services like NextDNS, ControlD etc. (nearly same blocklists enabled) the issue is not existent

It looks like it's something correlated to AdGuard DNS service at all?

pictosun commented 5 months ago

I changed the DNS of fritz.box to 1.1.1.1 (one.one.one.one DoT within) and on all my devices back to standard (away from 1.1.1.1 back to automatic) and the error just came back on my iPhone in this case. (iPad is doing well - but I just started testing).

Chinaski1 commented 5 months ago

@pictosun Hello there!

We're monitoring the situation but unfortunately we can't reproduce the issue on our end.

pictosun commented 5 months ago

We're monitoring the situation but unfortunately we can't reproduce the issue on our end.

Thanks for your feedback. What I can say for sure is, that it's not the mobileconfig itself because it also happens on systems with dnsproxy installed and also when using DoQ instead of DoH.

And it also looks like it is not some blocklist itself. Must be something different.

Would be nice to get some feedback from @emeritaacuity0u as he is also having those issues.

ghost commented 5 months ago

I don't have anything more to add at this stage. When using any AdGuard DNS server, my iOS App Store has connection issues intermittently. When I use another companies DNS server, it works.

pictosun commented 5 months ago

@emeritaacuity0u which Blocklists do you use?

ghost commented 5 months ago

AdGuard DNS filter.

The issue isn't happening with ControlD, or NextDNS for me.

pictosun commented 5 months ago

So it's definitively not the filter list at all. We do use completely different lists.

@Chinaski1 Maybe any advice what we both can do to troubleshoot more? What about the SERVFAIL fallback to CF and Google - could this maybe causing the issue for users?

pictosun commented 5 months ago

Could it be maybe something with EDNS (ECS ) implementation. When changing WAN networks (IPv4/IPv6) and maybe jumping to another EDNS (ECS) that the requests to some service (Apple App Store etc.) are somehow corrupted and you don't get correct responses?

Just an idea.

ameshkov commented 5 months ago

Let's try at least narrow it down to either filtering issues or the DNS server issues.

Please try using AdGuard DNS Unfiltered configuration (unfiltered.adguard-dns.com), if the issue can be reproduced with it then the culprit is filtering (filter list, access settings, etc, there are many things that change the normal DNS resolution).

If it's reproduced with AdGuard DNS Unfiltered, then we'll be looking from that side.

And one more thing. I guess we're at a point when to troubleshoot we need to look at the device logs.

pictosun commented 5 months ago

Hi @ameshkov

I will try it. You mean generally using unfiltered.... as DNS mobileconfig for the devices and the router?

What about device logs? Which logs do you mean in general so that I can investigate.

pictosun commented 5 months ago

AdGuard DNS filter.

The issue isn't happening with ControlD, or NextDNS for me.

@emeritaacuity0u Can you see any changes? For me it looks like it is getting a bit better (not that often blocked like it was a week or two ago).

With ControlD, NextDNS it still 'never' happens at all.

pictosun commented 5 months ago

Just a short update. Today it started to get really bad again. As it is not always the same I do believe that it is definitely something ad AdGuard side, what is bringing up those issues.

@ameshkov What could it be, that the last days it was quite ok and with this (morning) it started to behave really bad and I'm getting more issues than the days before?!

ameshkov commented 5 months ago

I will try it. You mean generally using unfiltered.... as DNS mobileconfig for the devices and the router?

The same way you setup the normal DNS.

What about device logs? Which logs do you mean in general so that I can investigate.

If it's a macOS device there's a "Console.app" that has all the device logs. It should have the necessary info.

pictosun commented 5 months ago

If it's a macOS device there's a "Console.app" that has all the device logs. It should have the necessary info.

@ameshkov Today I had those issues on my macOS machine and started console. Did some filtering of all the entries and could find, that there are some issues before the connection got working again. Don't know if any of those correlate to the issue but here are some:

Hope some of the messages can help to troubleshoot. Unfortunately I only have very limited knowledge, but I would still like to make a guess...: Could it be that it is because macOS performs/expects a certain caching here and if you open the App Store, for example, and the same server does not respond as stored in the caching (due to the dynamic change of the AdGuard DNS servers), that problems then occur as described? See #790

hagezi commented 5 months ago

@pictosun I think it's an iOS steering problem, you've tested it yourself. If you use 1.1.1.1 as the system-wide default DNS in iOS instead of your router (FritzBox) and also use the AdGuard DNS mobileconfig, the problem does not occur according to your statement.

Which FritzOS version are you using? The version prior to Fritz!OS 7.59 had problems with DNSSEC in combination with DoT.

You should test the following again. Use the FritzBox as system DNS as usual. In the FritzBox you use 1.1.1.1 as legacy Internet DNS not via DoT!

As I said, we operate countless Apple devices here in our immediate and wider circle of family and friends with AdGuard DNS and nobody has these problems.

pictosun commented 5 months ago

@hagezi Thanks for your feedback. Will give it a try. It does happen for iOS and macOS and 'only' when using AdGuard DNS (not with NextDNS and ControlD). As far as I understand your explanation the issue also should occur when using any other provider, or am I wrong?

And the other thing is, that it also happens when on the go via mobile network (and no fritz.box - hopefully - at provider side).

@emeritaacuity0u Do you have a fritz.box within your network scenario when the issue occurs?

ghost commented 5 months ago

7590AX on FRITZ!OS 7.81. I wont be able to try it out anytime soon.

As the issue happens both on and off the home network though, the router is not my single point of failure.

pictosun commented 5 months ago

Thanks for your feedback.

So it must be something else what is causing those issues.

pictosun commented 5 months ago

@hagezi After some reading I tried another approach... I read somewhere, that errors can occur, when using a DoT profile with a number at the 'beginning' of the URL. So I tried to make a new profile (mobileconfig) starting with a letter. And will check, if this will work better? Don't know if this is true (I can't really imagine) - but I will give it a try.

Another point is maybe what I asked within #792 - and forgive me if I'm wrong with this. I'm not an expert - I try to help with my means as best I can.

pictosun commented 5 months ago

And will check, if this will work better? Don't know if this is true (I can't really imagine) - but I will give it a try.

Doesn't help. The issue occurred again and again.

So sad to see. Today I made some research and found out, that @emeritaacuity0u and myself are not alone. But they're not reporting it to Github. Don't know, if they do switch to another provider or just ignore the issues.

Of course I'm now trying to click Apples App Store more often, than before. But it really is annoying to see all those outages while using AdGuard DNS.

And as I already said before it must be something on AdGuards end, as it really never happens with any other DNS provider (ControlD, NextDNS, native provider, Cloudflare, dnsforge and many others).

It really looks like AdGuard is doing something different in some way but I don't know what and why this makes it so unreliable (at my side).

ameshkov commented 5 months ago

And as I already said before it must be something on AdGuards end, as it really never happens with any other DNS provider (ControlD, NextDNS, native provider, Cloudflare, dnsforge and many others).

Do you set them all up using a DNS profile?

In general, is it true that the problem only happens when you set up AdGuard DNS using a .mobileconfig file?

If and only if this is true, we may try specifying the IP addresses in the mobileconfig file. You can create such a file yourself, just replace xxxxxxx with your device ID in the content below.

``` PayloadContent DNSSettings DNSProtocol HTTPS ServerURL https://d.adguard-dns.com/dns-query/xxxxxxx ServerAddresses 94.140.14.49 94.140.14.59 PayloadDescription Configures device to use AdGuard DNS PayloadDisplayName AdGuard DNS Personal PayloadIdentifier com.apple.dnsSettings.managed.64a1d190-a00a-4d81-92e8-33a37b255df6 PayloadType com.apple.dnsSettings.managed PayloadUUID 29bbe037-a2d0-4d10-a2c5-3111fb99f9a4 PayloadVersion 1 PayloadDescription Configures device to use AdGuard DNS PayloadDisplayName AdGuard DNS Personal PayloadIdentifier 15ca3ab0-91dc-43ef-ab02-4a3cbe64385d PayloadRemovalDisallowed PayloadType Configuration PayloadUUID 38cc616a-3d9b-4aba-a501-ee31dfd82a08 PayloadVersion 1 ```
pictosun commented 5 months ago

Do you set them all up using a DNS profile?

In general, is it true that the problem only happens when you set up AdGuard DNS using a .mobileconfig file?

@ameshkov Thanks for your feedback. In earlier times I also tested DoQ and DoT for macOS (with Little Snitch 6 and it's new DNS proxy integration) but cannot remember if it also happens there or not as the macOS systems don't move that often to other WAN networks.

Concerning the mobileconfig:

But when checking your .mobileconfig created from the dashboard it looks like I found a few small bugs.

Don't know if this is causing the (or any) issues? Overall it looks like a bug which should be corrected in general. mobileconfig_01 mobileconfig_02

pictosun commented 5 months ago

@ameshkov Concerning the bugs within mobileconfigs should I open a separate bug issue or is it ok, when leaving it over here as you will solve the issue internally?

ameshkov commented 5 months ago

What do you use to view the internals of a signed mobileconfig?

I don't think a mobileconfig with the highlighted errors can be installed in general as it won't be a valid XML. Could it be that the viewer that you're using is distorting them somehow?

pictosun commented 5 months ago

What do you use to view the internals of a signed mobileconfig?

I'm using some kind of editors (CotEditor, BBEdit and so on). The important part of those files is always readable.

If I open it with CotEditor in this case it shows: 01 02 03

So different but also not regular character set.

And as I already opened/edited and worked with many .mobileconfigs and never have seen such inconsistencies within this part of this configs I thought, that this could be a bug at all.

ameshkov commented 5 months ago

My point is that with the errors that the editor shows macOS/iOS wouldn't be able to import the file so most likely the problem is with the editor.

pictosun commented 5 months ago

I didn't edit the files. I was just looking at them to find maybe in error because of the issues I do have....

Isn't it strange that nearly all of those entries are correct, and only 2-3 are wrong. So I thought there is something wrong within automatic .mobileconfig creation at all.

ameshkov commented 5 months ago

I am saying that the editor fails to read the signed content properly.

Anyways, have you tried a mobileconfig with server addresses specified? Does it help?

pictosun commented 5 months ago

Anyways, have you tried a mobileconfig with server addresses specified? Does it help?

Cannot say for sure. I'm still testing. One config with IPs and the other one with hostnames - both are running at the moment. The problem doesn't exist all the time - looks like it sometimes happens more often than other times...

So it's not that easy to find out what's the issue at all.

pictosun commented 4 months ago

Just to give a feedback, that it also happens on systems using DoQ and DNSproxy (like LittleSnitch is offering). So it's not a mobileconfig issue at all.

Happened a few minutes ago again... After a special amount of time it starts working again and the first entries within the logs are requests to CDNs (but that's normal I think because of Apple using CDNs for their App Store). But maybe that's something to put in consideration when finding out the issue?!

pictosun commented 4 months ago

And another short update... Next device is also having issues. I think it could be something correlated to #795 as I'm seeing huge spikes there when checking TCPPing, DNS probes etc.

pictosun commented 4 months ago

And the next device is getting those errors... At the moment a lot of issues.

pictosun commented 4 months ago

@emeritaacuity0u Any update(s) from your side?

ghost commented 4 months ago

No, I do not have time to test.

ameshkov commented 4 months ago

Well we tried every indirect way to figure out what's happening and I guess at this point the only thing that can answer this question is the device logs exported from console app + the exact time when the issue happened + a detailed explanation of what was happening at that time (how did you understand that something is wrong and what happened next).

pictosun commented 4 months ago

Today a lot of issues appeared again.

As mentioned a few posts earlier it could really be something because of the errors within automatic .mobileconfig configuration profiles when downloading them via dashboard.

See the error message within console.

SCR-20240714-rblh
pictosun commented 4 months ago

@ameshkov Enclosed some more details to my latest post. It really looks like the issue is coming from your side (somehow).

ameshkov commented 4 months ago

A couple of lines without other details don't help much, sorry.

We need three things to troubleshoot, not having one of three makes troubleshooting impossible.

  1. Issue description (what happened, what error have you seen in the UI)
  2. Exact time of the issue
  3. Logs exported from console app (hand-picked lines are not enough since we don't know what we're looking for yet). I understand that sharing the whole log file might be too much, but can you at least export a 2-3 minutes period that corresponds to the time when you experienced the issue?
pictosun commented 4 months ago

Thanks for your feedback.

The point is, because of privacy reasons I cannot send you those logs as it's too much private information inside.

What's happening all the time is that I cannot connect to Apple services when using your DNS services within my setup. (see screenshot). When reconnecting to the network(s) it does work again (don't know if it's a caching issue or something else?). And it does only happen when using AdGuard DNS services....

Left is Apple Music, right is Apple App Store

I have an idea: I think I will try now a .mobileconfig using the same DoH setup as your generated .mobileconfig one and doublecheck if it is working without any issues... (will use one generated by https://dns.notjakob.com) But overall I don't think it will solve the issues as I already tried it via DNSproxy and LittleSnitch DNS configurations.

But let's see if I see any difference.

SCR-20240714-qvkk
Wrestor commented 4 months ago

I don't know if this is mentioned in this thread before, but this issue affects every platform, the AdGuard dns profile is installed to (iOS, iPadOS, macOS & visionOS). On my end, even though I excluded my WiFi SSID, this bug still affects me!

Note: I have always had my router set to the public AdGuard dns (ipv4 & 6), it isn't affected by this bug.

pictosun commented 4 months ago

@Wrestor Did you install the .mobileconfig from AdGuard DNS Dashboard or created one by yourself? Does the issue happen all the time at your side (even if it is difficult to control it all the time).

Wrestor commented 4 months ago

@pictosun I'm using the Default Server configured via "Method No2: Configure AdGuard DNS manually" > iOS > open profile constructor. DoH & entered my wifi ssid to disable it on my network, as i am already using my router configured with their dns(Default server as well).

pictosun commented 4 months ago

@Wrestor Thanks for the feedback.

Is it possible for you to try out some stuff. For me I'm testing an self created mobileconfig and since yesterday it does work well without any issues (but it's only a small amount of time). You can create them for yourself via the entry above from @ameshkov or for example via this one https://imazing.com/profile-editor or https://dns.notjakob.com

Will test a bit longer and when everything is doing will it looks like the AdGuard DNS dashboard profile creator does have some issues (but at the moment it's to early to say).

Maybe you can also jump and test so that I'm not alone here.

pictosun commented 4 months ago

Short update from my side. Those manually created mobileconfigs don't help. The issue is still there.

Looks like I have to change the provider as it's happening all the time for every client within my family. 😢

ghost commented 4 months ago

Unfortunately I've had to do the same. The issue isn't isolated to the mobile config's and happens when you use the AdGuard for iOS app with either native or AdGuard DNS implementation.

I've used DoT, DoH2, and DoH3 with AdGuard Personal DNS with the same outcome.

pictosun commented 4 months ago

So let's hope someone somehow finds out what those issues are... @emeritaacuity0u @Wrestor are you still 'on board' or already jumped to another provider?

pictosun commented 4 months ago

@hagezi and/or @ameshkov Could it be also be some weird setting within iCloud Private Relay, Hiding IP Addresses from Trackers, Prevent use advanced tracking and fingerprinting protection or something else?

I've seen within your knowledge base at the website, that you describe a few settings there, but as far as I know @hagezi is using some of the settings?

Gerd, how did you manage those settings in detail?

And overall en- or disabled the setting "Block access to iCloud Private Relay" within AdGuard-DNS Dashboard?

Maybe that's the reason of all those issues?

ameshkov commented 4 months ago

I am not entirely sure if iCloud Private Relay is used by iTunes app tbh. Do you have it enabled?

pictosun commented 4 months ago

No - I don't have it enabled, but there many other settings which could also be some kind of influencing iTunes and especially Mac/iOS App Store where the issue occurs...

Maybe @hagezi can jump in an say how he has configured all those settings as mentioned above. So it's easier to troubleshoot from my side as there are too many combinations to test.

hagezi commented 4 months ago

You have written that you have disabled these options, so it cannot be the cause of your problem. For me, it also makes no sense to play the combination lottery with these options.

Just do the test that has already been requested here, take the unfiltered AdGuard DNS as mobileconfig and see if the problem also occurs with it, see https://github.com/AdguardTeam/AdGuardDNS/issues/777#issuecomment-2180172987

Furthermore, according to one of your statements, it works if you use Cloudflare 1.1.1.1 as the system DNS instead of your router, for example, and at the same time a mobileconfig that uses an AdGuard DNS profile. The steering (the resolution of the dns name from the mobileconfig) then takes place via the Cloudflare DNS and not via your router and the upstream DNS configured there. You could then search further in this area, but everything has already been said.

We're going round in circles here ...