search

AdguardTeam / AdguardForAndroid

Open bug tracker for Android version of AdGuard.
https://adguard.com/
1.32k stars 90 forks source link

DNS Filtering does not work with HTTP root proxy in some cases #2977

Closed TheHasagi closed 1 year ago

TheHasagi commented 5 years ago

Regarding #2975

DNS Filtering isn't working with HTTP root proxy mode (auto). DNS User filter, DNS Custom filters are also affected by this issue, in the VPN filtering method, there are no such problems. Problem is reproducing with IPv6.

Preconditions:

The device should have root access.

Steps to reproduce

  1. Open AdGuard - Settings - Network - Filtering method - tap on Local HTTP Proxy
  2. Turn on DNS filtering - Select any DNS server
  3. Open filtering log and check for any DNS requests

Expected behaviour

DNS requests must exist

Actual behavior

There are no DNS requests in the filtering log

Your environment

ameshkov commented 5 years ago

@TheHasagi this looks like a serious issue so I've reassigned it to v3.2.

Please test it again and make sure that this is not related to IPv6 (see #2885)

ameshkov commented 5 years ago

@admitrevskiy please take a look tomorrow

TheHasagi commented 5 years ago

It is maybe related to the Private DNS option. Waiting for data.

ameshkov commented 5 years ago

Moved to v3.3 as this is not a new issue.

So there are three possibilities when it might not work, and we should handle at least two of them:

  1. When Private DNS is enabled (even in automatic mode), some apps (like Chrome) start using it.
  2. When the DNS server is in the private network, it is automatically excluded.
  3. IPv6 DNS server (see #2885)
patrickdrd commented 5 years ago

as I reported on my own issue, it gets broken after filters update I think, not necessarily dns filters update though

patrickdrd commented 5 years ago

it still doesn't work on the latest nightly.. after filters update dns filtering didn't work, please fix it because http proxy mode is unusable without dns filtering..

artemiv4nov commented 4 years ago

@admitrevskiy we should think about changing the DNS IP address and update it, also.

admitrevskiy commented 4 years ago

Done.

Testing instructions for QA:

1)- Enable Local HTTP Proxy auto mode and DNS filtering

2) - Enable private DNS in Android settings and DNS filtering in AG settings

3) This one is hard to test. You need IPv6 network and device with broken ip6tables

patrickdrd commented 4 years ago

unfortunately, it's not fixed yet :( and it's a shame we rooted users have to wait for so long in order to use the http proxy method, which is way better that the vpn one..

anyway, I repeat, I don't even have android 10, just 2 phones with nougat, I switched to http proxy after you said you fixed it, 3-4 days passed with a couple of dns filter (custom/user) updates and bang, dns filtering all of a sudden dead!! once again a total disappointment, I don't know what else to say :(

TheHasagi commented 4 years ago

@patrickdrd

Just to clarify - is this happening when you're connected via IPv6? To be sure please, follow these steps:

  1. Collect the debug log as it's explained here.
  2. Remember the exact time when the issue was reproduced, we will need it to find the corresponding records in the log file.
  3. Additionally, capture the system bug report after reproducing the issue. Here's the instruction.
  4. Send both files to devteam@adguard.com. Mention the Github issue number and the exact time when the issue was reproduced.
patrickdrd commented 4 years ago

I don't know about ipv6, I'm avoiding it whenever I can, anyway, dns filtering doesn't work at all when this happens, in all cases, I don't know what else I can say to explain it.. also, I cannot help you in debugging because as I've said dns filtering breaks after 3 or 4 days - this is very random and because I want software that works I immediately run to switch to the vpn method and I don't want to switch again, sorry, maybe you should debug it yourselves guys, I've got to accept that dns filtering in http proxy doesn't and I doubt if it will ever work in adguard..

TheHasagi commented 4 years ago

@patrickdrd

Unfortunately, we cannot guarantee that DNS will work with IPv6 in HTTP proxy mode (root). We are working on a solution.

patrickdrd commented 4 years ago

I don't think my issue is related to ipv6, because as I've said it's generic, dns filter doesn't work in any app, any request, it goes completely broken.. anyway, since I'm desperate to use http proxy and I'm tired of waiting, I installed adaway and I let it do the dns filtering with the hosts file, which works (I didn't enable any ipv6 blocking in its settings), since adguard is incapable/inconsistent in doing so..

TheHasagi commented 4 years ago

@patrickdrd

To troubleshoot the issue you are facing at the moment with DNS filtering we need debug logs with a reproduced issue. So please, follow the instructions above when the problems will return again.

patrickdrd commented 4 years ago

no, sorry, I've done that in the past without any fix(es), for now I'm fine with adaway, I don't have time or effort to debug your app, as I've said (again...) you've got the resources (I assume) to troubleshoot your app yourselves, we users want an app that works, me at least, no time for beta testing, I'm a paid user (not a beta or a cracked app user), so I want and I need an app that works

patrickdrd commented 4 years ago

ok, I'm talking with artyom ivanov @artemiv4nov on telegram

patrickdrd commented 4 years ago

after doing some testing I'm noticing that some rules work at first (dns), while others take some time (5-10 minutes or so) to catch up (after updating the dns filter), how do you load them? sequentially? maybe this is the issue in my case..

actually I found out from the filtering log that after a dns filter update, is takes too long for the http proxy to start blocking some requests again.. In the case I noticed the filter was updated at 12:12 and http proxy started blocking 12:25 again, I hadn't witnessed any such thing in the past, using either adguard's vpn or adaway or even dns66, I don't know if this "caching" (as described by @artemiv4nov) is due to android or your implementation of http proxy, but I know that it causes problems in http proxy, so maybe I'm better of using adaway to block my hosts and adguard for the rest of the filtering (since I'm rooted of course)?

patrickdrd commented 4 years ago

any news? is anyone working on this?

artemiv4nov commented 4 years ago

@patrickdrd , Hi! I and @admitrevskiy are working on your solution. I pay attention that your issue is difficult because of System DNS cache on Android.

We analyzed the Adaway decision and came to the conclusion that our methods are different, and we can not compare what works better for them. They just have a different way.

So, we'll try to find the solution for DNS resetting as far as possible.

patrickdrd commented 4 years ago

adaway works by blocking traffic through the system/etc/hosts file, which obviously is different than your implementation, however, are you sure that this is a "cache" issue, because sometimes it works fine, while others it takes a while to start blocking again.. if the requests were cached they would be cached all the time, because I open the tapatalk app many times a day, I think this sort of inconsistency is due to the http proxy (server?) not working properly all the times..

artemiv4nov commented 4 years ago

@patrickdrd You can re-check it following by steps: 1) Start AdGuard protection 2) Update your custom DNS filter 3) Be sure that part of rules (or all rules) are broken and AdGuard doesn't use it for DNS filtering 4) Change network (for instance, from Wi-Fi to your mobile network) 5) DNS cache for all applications and the system should be reset and your custom DNS rules work again.

Also, we will be very grateful if you send us the log after these steps.

patrickdrd commented 4 years ago

what do you mean by that? "DNS cache for all applications and the system" isn't reset after a filters update? also, I didn't change network (from wifi to mobile data) to any of the tests I did before, because dns filtering was broken, even in the same network anyway, if you think/saw that this is the case, can you force dns cache to be cleared after a filters update? maybe you should be doing this regardless of this issue, because what if the user wants to block a request already in the cache?

artemiv4nov commented 4 years ago

We can reset the DNS cache only for our application (so, we did it all time) But other apps have own caches that we cannot reset (Other apps' cache resets after the network changes).

patrickdrd commented 4 years ago

yes, but network wasn't changed in all my tests/cases also, is dns cache so inconsistent in apps? I mean some times it works immediately, some others it takes 13 minutes?

artemiv4nov commented 4 years ago

That's why I suggest you re-check the bug because changing the network after the DNS rules stop working should solve the problem. Every application can keep its own DNS cache, so if we reset our DNS cache, Chrome's cache (for example) has been not reset.

I mean some times it works immediately, some others it takes 13 minutes?

It all depends on whether the applications have time to send requests while the protection of AdGuard is restarted (at that time the regular DNS is working, not ours) or not.

patrickdrd commented 4 years ago

but the problem isn't with chrome, but with the tapatalk app (for example).. also I don't want to change the network, because one thing I love about the http proxy is that I avoid the reconfiguration each time network or connectivity changes, etc, while if I do that with vpn, many things break (https://github.com/AdguardTeam/AdguardForAndroid/issues/3189),

and last but not least, application (tapatalk in particular here) isn't sending requests in background, the requests I'm interested in (tapatalk logos) load whenever I load the app and only then.. and I load the app only after adguard finishes loading its updates

patrickdrd commented 4 years ago

one more thing: forget about adaway, when I was using adguard vpn mode I've never had such issues, so, no,I don't agree, it's not a dns issue but something is wrong with http server implementation

admitrevskiy commented 4 years ago

VPN is recognized as new network connection, which automatically clears the network cache, while HTTP proxy mode does not trigger a network change. I've tried triggering cache cleanup with # adb shell am - a $android.net.conn.CONNECTIVITY_CHANGE ... but it looks like the issue remains unresolved. We're still looking for a solution to this problem.

mnrivera210 commented 4 years ago

Wanting to use Adguard with a VPN is impossible due to this bug. Any progress on a fix?

sebastianjena commented 3 years ago

Still not fixed, just happened to me on 3.6.2 RC1, rooted Android 11

artemiv4nov commented 3 years ago

It can't be fixed quickly because we can't clean the cache for each application

patrickdrd commented 3 years ago

meanwhile, latest updates "Fixed] hepsiburada.com - HTTPS filtering issue #1406" break filtering on all apps on android 7 rooted, really terrible job, that's why I'm sticking with .50 version, besides that.. nah.. ugly result really.. apps don't load or load very slowly etc

ameshkov commented 3 years ago

@patrickdrd could you please be more specific? What exact version? Breaks what? Where? What device? How does it look like exactly?

patrickdrd commented 3 years ago

well I don't think you'll understand, because you don't know these apps, but anyway let's give it a try: on adm (advanced download manager) I click to download adguard nightly (I've got a shortcut saved on nova's launcher home screen with the adguard's nightly download url, well, since .51 nightly, the url doesn't resolve in order to proceed with the download, like it does with the .50 version!! also, images don't show on tapatalk app and sofascore app takes ages to load the matches, I think, as I've said the filtering for apps is broken somehow, I don't know what you're working on for the last 4 (!) nightlies (same changelog), but please look for side effects breaking already working functionality of the app! once again, I'm rooted with system store certificate, on nougat and I'm using the HTTP Proxy mode, same symptoms on two devices with the same configuration: a nokia 6 and a xiaomi redmi note 5a

ameshkov commented 3 years ago

@patrickdrd okay, we've not seen reports like that so we'd probably need to know more about it and read the log.

the url doesn't resolve in order to proceed with the download,

What exactly is that URL? What "doesn't resolve" mean, could you please show how the error looks like?

also, images don't show on tapatalk app and sofascore app takes ages to load the matches,

On a particular forum or somewhere else? Mind showing a screen recording?

Also, for this particular issue, it'd be interesting to get the debug-level log + timing (when the issue occurred).

patrickdrd commented 3 years ago

the url is https://static.adguard.com/android/nightly/adguard.apk and it's only an example of the damage that newest nightly updates do to apps like that, I mean advanced download manager editor inputs this url, but cannot retrieve information for it (if it's a site or a zip/apk to download) in order to proceed so all I'm getting is a circle looping and looping forever, I think I've fallen a couple of times already to cases such as this and I think apps are broken, I've got myself a new device android 10 and even though I rooted it, I'm forced to use a vpn (and browser only mode) for it due to these reasons, many apps misbehaving, but in this case I guess it's the android 10 to blame, so I shouldn't complain, I know...

also , sofascore is a live sports app, when the user opens it, he is presented with the matches of that day, well, with .51-.54 that matches list took a long time to load and the same happens with tapatalk images, i.e. images loaded inside the tapatalk app, I mean they load eventually but they take a lot of time

I also don't have the time to debug this app (once again), I mean I write, I develop and debug my own apps, why would I debug an app I paid for and I would like some support?

ameshkov commented 3 years ago

@patrickdrd you're a power user with a complicated configuration that is quite different from what 99% of users have. Regardless of whether you paid for the app or not, we need your help in debugging most of your issues. That's not because we're trying to shift some work to your side, we just cannot do the same ourselves.

Alternatively, follow the common procedure of reporting issues -- you see this template every time you open an issue in this repo. Besides all the information (and screenshots) that are requested there, we'll also need your settings backup + magisk settings backup.

artemiv4nov commented 1 year ago

@patrickdrd Good news! I implemented a new version of the Automatic proxy mode in the app. Now we can filter DNS IPv6 traffic. Please follow the 4.0 milestone.

patrickdrd commented 1 year ago

I tried the latest nightly for a bit (with auto proxy mode of course) and I noticed the AG restarts both when restoring connectivity (wifi I think) and when changing network (from wifi to mobile data and vice versa), why is that?

one of the advantages of auto proxy was that it did't require those restarts (as in 4.0.82), any chance you disablle them?

Rtizer-9 commented 1 year ago

@patrickdrd app restarting is I think getting tracked in another issue and there are also some low level configs in advanced settings to stop adguard reconfiguring itself on network changes which you can experiment with.

For me personally dns filtering is working fine in automatic proxy mode, it's the cosmetic filtering which doesn't work under some situations like while routing the data through another proxy.

patrickdrd commented 1 year ago

all you say are corrent but stand only for 4.0.82, have you tried latest nightly?

Rtizer-9 commented 1 year ago

@patrickdrd oh so the bug is back again; no I haven't yet updated to 4.0.585. I'm on 4.0.570 right now and with automatic proxy (not the manual proxy) dns filtering is still working. I'll update when I'll update (pun was definitely intended).

patrickdrd commented 1 year ago

@Rtizer-9 no, I meant I can't find an option to disable pausing the protection on .585 ("stop adguard reconfiguring itself on network changes" you said)

Rtizer-9 commented 1 year ago

I think the previous reconfiguring flags are combined and renamed as "restore VPN automatically" which you might wanna experiment with.

patrickdrd commented 1 year ago

hm, are you sure? what does vpn have to do with auto proxy? it doesn't make sense to me, if it so, maybe they need to rename it, otherwise tell us what they did to it

patrickdrd commented 1 year ago

https://i.imgur.com/qbyzjsr.png