Exclude patterns not working

z33ky commented 6 months ago

I have version 8.8 on Firefox 121.0. The extension page reports "No update found", even though the changelog here mentions a version 8.9.

I've seen a couple of issues stating issues with the 8.0 upgrade and patterns, but I might be too dumb to properly update my patterns. I can't get the simplest pattern to work.

[
  {
    "include": "exclude",
    "type": "wildcard",
    "title": "Plain Host",
    "pattern": "*://heise.de/",
    "active": true
  }
]

Result: https://heise.de/ is still accessed through the proxy. I tried *://*.de, *://*/, *, heise.de and a couple of RegEx versions as well. No success.

FWIW I don't even need exclude patterns. I have a couple of patterns, that should be accessed via proxy, and the rest of the Internet should be accessed without. That previously worked by me just setting a couple of "Include" patterns, I'm not sure if the new version recognizes this means exclude-by-default, so I'm trying to setup a pattern for that. I've configured this fresh proxy to test exclude-patterns though, so there are no other settings that interfere, but as stated above I am unable to get even the simplest pattern to work and need help. Putting the site in the "Global Exclude" list works, but I don't believe I can work with this setting for my actual configuration that I explained.

erosman commented 6 months ago

I have version 8.8 on Firefox 121.0. The extension page reports "No update found", even though the changelog here mentions a version 8.9.

v8.9 is the dev version i.e. beta and not released yet.

I can't get the simplest pattern to work.

That pattern looks fine. You can always use the "Pattern Tester" to check if a pattern matches.

My guess is that another pattern is causing it. If you post your settings (remove private data), we can try an see what is happening.

I'm not sure if the new version recognizes this means exclude-by-default

Sure. Everything is excluded (no proxy) by default. Only the patterns that users set, is proxied.

z33ky commented 6 months ago

That pattern looks fine. You can always use the "Pattern Tester" to check if a pattern matches.

Okay, the pattern passes according to that page. So you are probably correct assuming another pattern (or setting?) is at fault, however I have no other pattern in the test profile.

Sure. Everything is excluded (no proxy) by default. Only the patterns that users set, is proxied.

I see. Even if I deactivate all patterns, it still goes through the proxy.

If you post your settings (remove private data), we can try an see what is happening.

{
  "mode": "disable",
  "sync": false,
  "autoBackup": false,
  "showPatternProxy": true,
  "passthrough": "",
  "container": {
    "incognito": "",
    "container-1": "",
    "container-2": "",
    "container-3": "",
    "container-4": ""
  },
  "commands": {
    "setProxy": "",
    "setTabProxy": "",
    "quickAdd": ""
  },
  "data": [
    {
      "active": false,
      "title": "test",
      "type": "http",
      "hostname": "redacted",
      "port": "8888",
      "username": "redacted",
      "password": "redacted",
      "cc": "",
      "city": "",
      "color": "#cc0000",
      "pac": "",
      "pacString": "",
      "proxyDNS": true,
      "include": [
        {
          "type": "wildcard",
          "title": "all URLs",
          "pattern": "*://*/",
          "active": true
        }
      ],
      "exclude": []
    },
    {
      "active": true,
      "title": "Intranet",
      "type": "http",
      "hostname": "redacted",
      "port": "8888",
      "username": "redacted",
      "password": "redacted",
      "cc": "",
      "city": "",
      "color": "#ae115d",
      "pac": "",
      "pacString": "",
      "proxyDNS": true,
      "include": [
        {
          "type": "wildcard",
          "title": "all URLs",
          "pattern": "*://*.redacted.local/",
          "active": false
        },
        {
          "type": "wildcard",
          "title": "all URLs",
          "pattern": "*://*.redacted.local:*/",
          "active": false
        }
      ],
      "exclude": [
        {
          "type": "wildcard",
          "title": "Plain Host",
          "pattern": "*://*/",
          "active": true
        }
      ]
    }
  ]
}

Thanks for the quick response.

erosman commented 6 months ago

Even if I deactivate all patterns, it still goes through the proxy.

How did you check that?
Is that what the log shows?
Have you refreshed the page? (to clear the cache)

Also....

Is your proxy server local?
Is it HTTP server or socks?
Why are you excluding everything? (i.e. "*://*/")

Exclude takes priority over include.

In any case "mode": "disable", mean no proxying. Refresh the tab that you are checking and see.

z33ky commented 6 months ago

How did you check that?

The site becomes unreachable if I select the profile in the extension. It is available when I disable the profile.

Is that what the log shows?

Yes, that as well:

CONNECT   Jan 04 12:26:32.691 [5024]: Request (file descriptor 5): CONNECT heise.de:443 HTTP/1.1
NOTICE    Jan 04 12:26:32.694 [5024]: Proxying refused on filtered domain "heise.de"

(from the proxies logfile)

Have you refreshed the page? (to clear the cache)

I did a "regular" refresh. Hitting Ctrl+Shift+R (or selecting "Disable cache" in the dev-tools) produces the same result.

Is your proxy server local?

It runs on the local network, but on another machine. The redacted hostname is a LAN IP address. Probably not actually private data...

Is it HTTP server or socks?

HTTP (tinyproxy).

Why are you excluding everything? (i.e. ":///")

That was a remains from my attempt to fix this situation. Removing that pattern (and activating or deactivating the include patterns) does not resolve the issue.

Visiting the .local sites works fine when the profile is active, i.e. the requests are routed to the proxy - even if the include-patterns are not active!

erosman commented 6 months ago

Can you check the FoxyProxy Log to see if Firefox proxying or not?
Is any other extension installed that deals with the proxying?
Is the FoxyProxy icon in color or grey?
What does about:preferences -> Network Settings say?

z33ky commented 6 months ago

Can you check the FoxyProxy Log to see if Firefox proxying or not?

No profile active (the .local URL fails to load)	Time	Method	Document URL
14:52:26	GET	http://foo.redacted.local/
...
14:52:23	GET	https://www.heise.de/	https://www.heise.de/icons/common-icons.svg
14:52:23	GET	https://www.heise.de/	https://www.heise.de/assets/styles/fonts/ss3vf.c70.ltc.woff2
14:52:23	GET		https://www.heise.de/
14:52:23	GET		https://heise.de/
14:52:23	GET		http://heise.de/

Intranet profile active (only the .local URL loads (since heise.de is rejected by the tinyproxy configuration)	Time	Method	Document URL	URL	Proxy Title	Type	Hostname
...
14:56:01	GET	http://foo.redacted.local/	redacted	Intranet	HTTP	192.168.178.116	8888
14:56:00	GET		http://foo.redacted.local/	Intranet	HTTP	192.168.178.116	8888
14:55:59	GET		https://www.heise.de/	Intranet	HTTP	192.168.178.116	8888

This is the configuration used:

[
  {
    "include": "include",
    "type": "wildcard",
    "title": "all URLs",
    "pattern": "*.redacted.local",
    "active": true
  },
  {
    "include": "include",
    "type": "wildcard",
    "title": "all URLs",
    "pattern": "*.redacted.local:*",
    "active": true
  }
]

Is any other extension installed that deals with the proxying?

No.

Is the FoxyProxy icon in color or grey?

It's in color: foxyproxy

What does about:preferences -> Network Settings say?

"No proxy"

erosman commented 6 months ago

Looking at https://github.com/foxyproxy/browser-extension/issues/112#issuecomment-1876940861, the 1st proxy "title": "test", has a catch all pattern. It shows as disabled i.e. "active": false, Is that the case?

Looking at the 2nd proxy "title": "Intranet",, it has an exclude all pattern i.e. "pattern": "*://*/",. Is that the case?

Can you post a minimal setting that can demonstrate the issue? Have you set anything in the system hosts file to resolve heise.de?

z33ky commented 6 months ago

Looking at https://github.com/foxyproxy/browser-extension/issues/112#issuecomment-1876940861, the 1st proxy "title": "test", has a catch all pattern. It shows as disabled i.e. "active": false, Is that the case?

Looking at the 2nd proxy "title": "Intranet",, it has an exclude all pattern i.e. "pattern": ":///",. Is that the case?

I have since reset the configuration ("Restore Defaults") and added the "Intranet" configuration once again, so it is the only profile available. It only has the two "include" patterns, no "exclude" patterns now.

Can you post a minimal setting that can demonstrate the issue?

You stated that everything is exclude-by-default. A profile with no patterns triggers this issue for me.

{
  "mode": "192.168.178.11:8888",
  "sync": false,
  "autoBackup": false,
  "showPatternProxy": false,
  "passthrough": "",
  "container": {
    "incognito": "",
    "container-1": "",
    "container-2": "",
    "container-3": "",
    "container-4": ""
  },
  "commands": {
    "setProxy": "",
    "setTabProxy": "",
    "quickAdd": ""
  },
  "data": [
    {
      "active": true,
      "title": "Intranet",
      "type": "http",
      "hostname": "192.168.178.11",
      "port": "8888",
      "username": "redacted",
      "password": "redacted",
      "cc": "",
      "city": "",
      "color": "#ae115d",
      "pac": "",
      "pacString": "",
      "proxyDNS": true,
      "include": [],
      "exclude": []
    }
  ]
}

That makes all requests go to the proxy. It does not seem to matter what I add in the patterns; includes, excludes, specific sites, all-matching pattern, the behavior always seems identical. I always "Save" the configuration, I've tried switching the profile to "Disable" and back to "Intranet" or restart the browser in between adjusting the settings, but once it's on "Intranet", everything goes to the proxy. If it's "Disable", nothing goes to the proxy.

Have you set anything in the system hosts file to resolve heise.de?

Nope. I assume you are unable to replicate my issue?

erosman commented 6 months ago

That makes all requests go to the proxy. It does not seem to matter what I add in the patterns; includes, excludes, specific sites, all-matching pattern, the behavior always seems identical.

Of course it does.

There are 3 types of modes in FoxyProxy:

Disable (no proxy)
Proxy by Patterns (proxy if matched patterns, and direct if not)
Individual Proxy (everything goes through this one, patterns are irrelevant)

You are now set to an individual proxy "mode": "192.168.178.11:8888", which means all requests go through this proxy.

ericjung commented 6 months ago

If you are using patterns, they are ignored unless FoxyProxy is set to “Proxy By Patterns” in the toolbar icon. But there seem to be other issues, too, like your proxy server not being able to load heise.de unless that is intentional.

z33ky commented 6 months ago

If you are using patterns, they are ignored unless FoxyProxy is set to “Proxy By Patterns” in the toolbar icon.

Okay. I thought it is a separate profile for some other feature that I am unfamiliar with. Selecting "Proxy By Patterns" works as expected.

But there seem to be other issues, too, like your proxy server not being able to load heise.de unless that is intentional.

That is intentional. Only URLs belonging to the intranet are allowed by the proxy.

I think this issue can be closed than. Thanks for the speedy help!

foxyproxy / browser-extension

Exclude patterns not working #112