Fix blocked tracker redirects

[Morku]

Hi, this is a feature request.

introduction AdGuard prevents us from Ads and Trackers. Thats great 👍 When pages not work correctly because of blocked tracks, the filter maintainer do a great job to exclude the tracker page for that specific page so that pages works correctly. As I can see, AdGuard Home got quickly really famous. It get's recommended as first line prevention against Ads and Trackers. Some pages (especially price checking pages or hotel/vacation booking) use a man in the middle tracker before bring the user to desination URL. This tracker is of course being blocked.

---

the problem Especially AdGuard Home (currently) can only block a DNS globally or allow for all pages, bringing User sometimes to a dead end. You could now allow the tracker and accept to get tracked again (which you want prevent with AdGuard) or you have to accept to find another way to destination.

---

examples

Let's buy a Nintendo Switch... :) and compare the prices:

https://www.idealo.de/preisvergleich/OffersOfProduct/6732882_-switch-neon-rot-neon-blau-neue-edition-nintendo.html

Example 1:

I think I want buy from LIDL, so I click the link: https://www.idealo.de/preisvergleich/Relocate/3325329051544986004.html?categoryId=3189&offerKey=dcb78009fa6e27ed273add668cee7ecc&offerListId=6732882-CE858F4B873D67F41E5F93B30943D080&pos=4&price=319.13&productid=6732882&sid=205547&type=offer

So you click it and you end up here: https://www.econda-monitor.de/link/st?emkd=2235083&pbid=1&advid=251&campaign=feed%2Fde%2F49133%2Fidealo%2F000000000100267084&target=https%3A%2F%2Flidl.media01.eu%2Fset.aspx%3Ftrackid%3D9A0006C4C1508D1F033FE2800009068D%26dt_url%3Dhttps%253A%252F%252Fwww.lidl.de%252Fde%252Fnintendo-switch-konsole-neon-rot-neon-blau%252Fp311463%253Futm_source%253Didealo%2526utm_medium%253Dpricecomparison%2526utm_campaign%253Didealo%2526mktc%253Dpricecomparison

It can't connect, because it is blocked in Adguard DNS.

If we check the URL now, we notice, that the destination URL is present: -> https://www.lidl.de/de/nintendo-switch-konsole-neon-rot-neon-blau/p311463?mktc=pricecomparison

Example 2:

https://www.idealo.de/preisvergleich/Relocate/3351815736271056163.html?categoryId=3189&offerKey=232adf18066ec1fd225b3db238286421&offerListId=6732882-F8B51A5B09F63E868F98FB8EDD18612C&pos=5&price=320.00&productid=6732882&sid=316003&type=offer

-> https://marketing.net.idealo-partner.com/ts/i5539833/tsc?smc2=l_4ZtUHHAUocLVyMFgEovg&smc3=3189&smc4=316003&smc5=idealo&rmd=3&trg=https%3A%2F%2Fshop.maingau-energie.de%2Fmultimedia-technik%2Fgaming%2Fnintendo-switch-neon-rot%2Fneon-blau-neues-modell-2019%3Fnumber%3D480041

• blocked because of marketing.net.idealo-partner.com

-> but destination in URL: https://shop.maingau-energie.de/multimedia-technik/gaming/nintendo-switch-neon-rot/neon-blau-neues-modell-2019?number=480041

---

Okay, let's forget the Switch. I can't buy it :(

Example 3:

Oh look, the Marvel collection looks nice here: https://www.kino.de/marvel/marvel-filme

https://gocp.stroeermediabrands.de/go.php?t=https%3A%2F%2Fwww.amazon.de%2Fdp%2FB075YLSBY5%3FlinkCode%3Dogi%26th%3D1%26psc%3D1&s=%2F%2Fwww.kino.de%2Fmarvel%2Fmarvel-filme&channel=website&contentPieceId=10027206&uuid=4e3bfb3c-3477-44b8-8382-1bb763d37587&site=KINO&deviceType=desktop

• blocked because of gocp.stroeermediabrands.de

Redirects to: https://www.amazon.de/dp/B075YLSBY5

---

Ok, time to go for vacation.

https://www.trivago.de/forward.php?ccid=Xi1cCrymtFFxBrK4Dq35NQAAACE&iRateAttr=0&freeCanc=0&bAlternativeHotel=1&mode=1&ctnr=1&gueRt=0&staRt=0&intent=0&enc2=b43c26190305012c86c27b2779db76793d41e8304fcdbac2ef16223afc46fa254505225952fc3c6a43901357b0e1f4c4154ce17f7cb702e2341347197000639c&csd=1&enc3=_zytpyDfcLtfLGSsSoSrX6q0ZxLcJGB5RfrANGXIDWfCIVUQClmMoOyCLCPR3siuKzSupVTcf.OCTBuRq-Czz3A6ErlNSGbyLuD7QOEWsm65lA1y0BwHOMbGdkAaLaZ-o9ENJcnOBLcQw55izwO6FXeuct2UjFzJ1AmbVRA%3D%3D&rid=6fe8884603045f6c581d231b7ce12d43&__typename=rsClcklB&group=1084&enc4=_bRArAA7JdR1uNQhVU7yfH1rixjfmHoNeJ8axeKjpc70zTWmWiJN3r9a05sUkSXlitigJXDY7f7fQ5YKzSn1ykSjsdhdDvUkuFE5mRJZk9judRqJ00vSvr1GuCnHvPsfArya4lsHrZLx9nJVHX0eeE-b-VaQT-a-EZJOJ5e4kp4wwTr0wrQ7clhZiEu9Zxq.hMz1oFUUYU2sL7jeEJNeh6.O2.L-QWh06l4YFzgz6r8o0RsIQVZK5K6s4ffcb9dqi6p1Gjx9-s01-cgCvPEu7e4Yzaghg5mRa.ELIH.oGUEOKfoLk4jlF6PMQDmHnV5-ZJI5bRzDCFMeGlD1JXg%3D%3D

https://ad.doubleclick.net/ddm/clk/252297487;126333561;l;u=hotelidMUCGIGI&u2=trivago_MUCGIGI_DE&u1=trb&cur=EUR?https%3A%2F%2Fsecure3.hilton.com%2Fde_DE%2Fgi%2Freservation%2Fbook.htm%3FinputModule=HOTEL_SEARCH&ctyhocn=MUCGIGI&arrivalDay=20&arrivalMonth=09&arrivalYear=2020&departureDay=21&departureMonth=09&departureYear=2020&numRooms=1&viewPackagesAndPromotionsRate=true&room1NumAdults=2&room1NumChildren=0&srpCodes=LV8&WT.mc_id=zINDA0DE1XX2OLX3DA4MS5Trivago6MUCGIGI7_%25epid%21

Oh hi Google. It's you again and you are blocked.

Would redirect to: https://www.hilton.com/de/book/reservation/deeplink/?inputModule=HOTEL_SEARCH

---

I would even go a step further and would like to see prevent "normal" redirect pages which is mostly Annoying and can also be part of tracking, like: https://adguardteam.github.io/AnonymousRedirect/redirect.html?url=http%3A%2F%2FGoogle.com No one likes to doubleclick a link and wait for the 'extrastep'.

---

the solution...?

AdGuard for Windows has an Extension engine and I would like to see an Extension based on the Trackerlist to fix the redirects directly to destination, without the need to allow all the Trackers. It should be based on a "known" list (like blocking Ads is based on) to prevent false positives.

In my opinion it is the next logical step to fix, what AdGuard "broke" :)

Please give me opinion what you think about the idea and if you like to do.

I already got so many calls from my family and friends because of this 'issue'.

There is an Extension for Browsers called "Skip Redirect" what is doing the job. The problem: this is based on algorythm and focused on specific wordings in URL. It does the job for my provided examples. But I encountered many false positives where Redirects in URL are necessary are needed to work. So in my opinion, the solution is only a maintained filterlist.

I know AdGuard is focused on blocking Ads, but I hope it is something you may think about.

Regards

[ameshkov]

But I encountered many false positives where Redirects in URL are necessary are needed to work.

Could you please provide a couple of examples? I haven't quite got the idea.

Regarding the general topic. It'd be quite problematic tbh. For some of these "redirect" services you simply cannot extract the target location from the URL and need to actually visit the page. And visiting is a problem by itself because it's already blocked pn the DNS level.

That's why we generally tend to not block redirect services on the DNS level - because in this case we simply cannot fix it after it's blocked.

[Morku]

@ameshkov

Here are false positive when you use the extension "Skip Redirect":

---

Go to https://www.google.com/imghp?hl=de&tab=wi&ogbl click on the little camera, enter a URL of an image.

Now Google loads e.g.: https://www.google.com/searchbyimage?image_url=https%3A%2F%2Faddons.cdn.mozilla.net%2Fuser-media%2Faddon_icons%2F520%2F520576-64.png%3Fmodified%3Dmcrushed&encoded_image=&image_content=&filename=&hl=de before a search result come up.

Now Skip Redirect see the redirect and will load https://addons.cdn.mozilla.net/user-media/addon_icons/520/520576-64.pnq?modified=mcrushed instead.

---

Other example. Reporting an issue for the filter maintainers. https://reports.adguard.com/de/new_issue.html?product_type=Win&product_version=7.5.1&url=https%3A%2F%2Fdeavita.com.html&user_agent=Mozilla%2F5.0%20(Windows%20NT%2010.0%3B%20Win64%3B%20x64%3B%20rv%3A79.0)%20Gecko%2F20100101%20Firefox%2F79.0&filters=11.14.1.2.3.4.6&userscripts=https%3A%2F%2Fuserscripts.adtidy.org%2Frelease%2Fadguard-extra%2F1.0%2Fadguard-extra.user.js&win.wfp=true&stealth.enabled=true&stealth.hide_search_queries=true&stealth.DNT=true&stealth.x_client=true&stealth.disable_third_party_cache=false&stealth.webrtc=false&stealth.push=false&stealth.location=false&stealth.disable_windows_telemetry=true&stealth.turn_off_advertising_id=true&stealth.disable_windows_defender=true&stealth.disable_wap_push_message_routing_service=true&stealth.flash=false&stealth.java=false&stealth.strip_url=true&stealth.block_third_party_auth=false

It discards all the necessary and useful information for the report and loads: https://userscripts.adtidy.orq/release/adquard-extra/1.0/adquard-extra.user.js because it's the last url in that URL

---

Third and fourth example are purchasing something from ebay and checkout to PayPal for payment. When the little PayPal window opens, I got an endless loading circle. And then the last is related to my workplace, based on a Salesforce platform. I wasn't able to open a Chat, because of the extension

---

I have asked the developer about the Google thing, but he does not fix page specific redirections (whitelisting). It's the way the extension work and nothing to blame here. But here I ask myself: Is this something I can install for my family and forget? No, I can't.

All the 4 examples are normal pages, useful information and no tracker pages.

The examples in my starting post on the other hand, are trackers who redirect to destination and here the extension comes handy, without loading the tracker page. Thats why I think a maintained list for blocked trackers would be useful.

For some of these "redirect" services you simply cannot extract the target location from the URL

I have seen such examples rarely. Most times, I can see the destination in the tracker URL. It would improve the situation a lot.

That's why we generally tend to not block redirect services on the DNS level

Not so for https://github.com/AdguardTeam/AdguardFilters/issues/60264 Exclude them yourself was the advise. Or https://github.com/AdguardTeam/AdguardFilters/issues/48633 You would need to allow doubleclick to redirect to destination page.

Regards

[ameshkov]

Here are false positive when you use the extension "Skip Redirect":

I meant the false-positives of AdGuard DNS -- what click-through domains do we block on the DNS-level?

Not so for AdguardTeam/AdguardFilters#60264 Exclude them yourself was the advise. Or AdguardTeam/AdguardFilters#48633 You would need to allow doubleclick to redirect to destination page.

Some domains are indeed multi-purpose and we can't unblock them (like doubleclick for instance). But if the domain is used exclusively as a click-through domain, we tend to not block it on the DNS-level.

Regarding "skip redirect"-like functionality in AdGuard itself, it seems that a browser extension is a better solution in this particular case and the reason for it is that the blocking is done on the DNS-level. In this case, AdGuard simply won't see the network connection because there is one. The browser extension would, though, see the attempt to make a request, and it can react to it and try to remove the redirect.

[Morku]

Fot the DNS tracker redirects, I myself could never say for sure, if they have a multi-purpose. So I won't request to unblock them and we would accept to track, even if there is (for this examples) no need.

If an extension is the better solution, why not. Would it be a good feature request for the AdGuard Extension? Should I raise one or you think it will have a bad chance? (Would be now 2 features of the Extension to support the standalone app, beside my request to filter HTTPS excluded pages only)

[ameshkov]

Should I raise one or you think it will have a bad chance?

Tbh I am halting every big feature of extension until manifest V3 migration is started (or not started). I am afraid that there's a chance that what we do simply becomes unusable in a year.

Would be now 2 features of the Extension to support the standalone app, beside my request to filter HTTPS excluded pages only

Tbh I still don't fully understand why would you need that, https exclusions don't mean that nothing is filtered, it just means that AdGuard does not try to decrypt connections with a particular domain. It's easier to simply clean the HTTPS exclusions list.

[Morku]

Tbh I am halting every big feature of extension until manifest V3 migration is started (or not started). I am afraid that there's a chance that what we do simply becomes unusable in a year.

At least, it's not a no 😉

In case we go Off-Topic here, but...

https exclusions don't mean that nothing is filtered

Exactly and so the pages starts to annoy with Cookie Banners I want to have blocked and rules exist in filters. After month I still not figured out, why excluding bank and payment pages is necessary for a local filtered Adblocker. If the information are to sensitive, then it would bne a discussion about why trust Adguard at all. On every shopping page I enter sensitive informations. Maybe I am wrong, because I have also heard it is to avoid errors, when original certificate is not in use. Yes, I had read the knowledge article, but I still don't get it. So I would prefer to have the list of https excluded pages accessable and in a readable format (with/-out www.) I can import to the Extension to not decrypt the connection, but have Adblocking on any page.

[ameshkov]

After month I still not figured out, why excluding bank and payment pages is necessary

Well, for some reason it makes some people feel calmer. This is indeed not necessary, and only in really rare cases it is done to avoid errors.

Actually, here's the full list of domains excluded to avoid errors: https://github.com/AdguardTeam/HttpsExclusions/blob/master/exclusions/issues.txt

[Morku]

Great, thank you for the list of the known issues pages. I will only add them. Then I can finally remove AdGuard Extension.

I can remove all pages with Ctrl+A and DEL, but I can't add all 55 pages at once (no Editor mode). I have to add them one by one, little painful... I know this is still a feature requested from other tickets. Is there any workaround outside the software possible? In which file they will be included, so I need only copy/paste the file for other systems?

And the skip tracker redirects might be something for AdGuard Assistant? :)

[ameshkov]

Is there any workaround outside the software possible?

Not really:(

And the skip tracker redirects might be something for AdGuard Assistant? :)

It might be indeed.

[Andy-Kl]

https://www.medipreis.de/preisvergleich/silymarin-stada-forte-hartkapseln-100-st-stada-consumer-health-deutschland-gmbh-13579384 ➜ Eurapon ➜ [bei diesem Shop kaufen] * https://www.arzneisucher.de/Preisvergleich/13579384 ➜ Eurapon ➜ [beim Shop bestellen]

[Morku]

Thank you for planing this feature now in https://github.com/AdguardTeam/CoreLibs/issues/1557