Add AdGuard's $network support on Firefox

Yuki2718 commented 1 year ago

Prerequisites

[X] I verified that this is not a filter list issue. Report any issues with filter lists or broken website functionality in the uAssets issue tracker.
[X] This is not a support issue or a question. For support, questions, or help, visit /r/uBlockOrigin.
[X] I performed a cursory search of the issue tracker to avoid opening a duplicate issue.
[X] The issue is not present after disabling uBO in the browser.
[X] I checked the documentation to understand that the issue I am reporting is not normal behavior.

I tried to reproduce the issue when...

[X] uBO is the only extension.
[X] uBO uses default lists and settings.
[X] using a new, unmodified browser profile.

Description

Documentation: https://adguard.com/kb/general/ad-filtering/create-own-filters/#network-modifier Related issue: https://github.com/AdguardTeam/tsurlfilter/issues/104

A specific URL where the issue occurs.

https://github.com/AdguardTeam/AdguardFilters/pull/160415/files

Steps to Reproduce

Check the links above

Expected behavior

NA

Actual behavior

NA

uBO version

1.51.0

Browser name and version

Firefox 116.0.3

Operating System and version

Windows 10

gwarser commented 1 year ago

Well it's possible to block at onHeadersReceived() time and the ip address information is available at this point

Will blocking on response suffice?

Yuki2718 commented 1 year ago

If can then display strict-blocking page.

DandelionSprout commented 11 months ago

Goodness knows how this would be implemented in practice, but $network is fantastic stuff when done right, especially to handle IPs known to host sometimes thousands of malware domains and only 0~2 legitimate domains.

JobcenterTycoon commented 6 months ago

This would also be useful to block rotating propeller adservers in real time.

We should also be able to add additional modifiers like ||127.0.0.1^$network,script,3p this would allow users to fine control their traffic which can't be setup with a regular firewall.

gorhill commented 1 week ago

Related commit (forgot to link issue in commit message): https://github.com/gorhill/uBlock/commit/c6dedd253ffbf649729a38998033c75993489859

JobcenterTycoon commented 1 week ago

Ok i tested on https://woxikon.in/ and the filter *$script,xhr,ipaddress=139.45.197.170 successfully blocked the Propeller request https://sswpawbv.xyz/

D4niloMR commented 1 week ago

It's possible to ipaddress block document? header too, would be good if possible.

gorhill commented 1 week ago

Real use case?

I did look at whether document should be blocked but it turned out to be a bit more work then I expected so I left this out.

D4niloMR commented 1 week ago

I was thinking about those domains from badware that have a lot of similar domains in the same ip https://viewdns.info/reverseip/?host=m8u9f3.com&t=1, https://viewdns.info/reverseip/?host=172.64.155.33&t=1

Though we would need to be very careful to not block legitimate sites.

DandelionSprout commented 1 week ago

I did some testing for 40min tonight in Firefox 130.0 with uBO 1.51.1b17. I don't intend to put immediate pressure on any involved parties, I'm simply listing them.

I was (with some difficulty) able to get $ipaddress= to work with IPv6 from as a proof-of-concept. The supported syntax is without square brackets, resulting in *$all,ipaddress=2a00:1450:400f:803::200e. Getting it to work on ipv6.google.com required more than one ping ipv6.google.com in PowerShell to see which IPs the many involved domains had, and getting it to work on IPv6+IPv4 sites (I tested with www.dr.dk) would be more than a little difficult.
Although $all is supported as a modifier, which results in e.g. *$all,ipaddress=199.232.42.217, the modifier does not seem to have any additional effects compared to having no additional modifiers.
Requests that can be blocked with $ipaddress include script, xhr, image, frame, css, and beacon (I was unable to test with media in this hurry).

• "I did look at whether [$doc] should be blocked but it turned out to be a bit more work then I expected so I left this out." Fair enough.

gorhill commented 1 week ago

Somehow I thought Firefox's ip field was using brackets for ipv6, turns out it's not. Will fix.

Fixed in 1.59.1b18

JobcenterTycoon commented 4 days ago

When i open https://720pflix.fit/ it shows the first propeller request as unblocked but i see no requests in the Firefox network logger. After a reload its fine.

gorhill commented 4 days ago

It's normal, the first time the IP address is not available in uBO's DNS cache, so the first call to onBeforeRequest() will log a non-matched entry. Once the the DNS resolution occurs, there will be another call to onBeforeRequest but with the IP address available, which will then match your filter. It works like this:

OnBeforeRequest
- IP address available?
  - Yes:
    - Add ip address to request details
    - Perform request matching (log entry)
  - No:
    - Fetch it through an asynchronous DNS query
      - DNS query fulfilled
      - Add ip address to request details
      - Perform request matching (log entry)
    - Perform request matching (log entry)

It's also how matching against cname has been working, there is always two entries in the logger when there is a cname. The logger really reports request matching calls, and there can be more than one. I suppose it would be useful to show at which stage the matching is made so that we know whether there was an actual request reaching the remote server, but there is not much horizontal space for more details in the logger.

uBlockOrigin / uBlock-issues