HTTP/409 (host header forgery) in Squid 5.7 since OPNsense 22.7.5

[marinbernard-pep06]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• [x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• [x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

OPNsense 22.7.5 bumps Squid from version from 4.15 to 5.7. Since then, our access log is filled with HTTP/409 errors. The cache log is also full of host header forgery warnings. Those of our proxy servers which are still running OPNsense 22.7.4 (with Squid 4.15) do not have such an issue.

HTTP/409 is thrown by Squid when the FQDN of the remote server does not resolve to the same IP address on the client and on the proxy server. This was a very common error with Squid 3, which was finally resolved with Squid 4, but is now reappearing with Squid 5. Like with Squid 3, this happens despite the fact that all our proxy servers and clients are using the same Unbound DNS server. To me, this is likely to be a upstream bug.

To Reproduce

Steps to reproduce the behavior:

1. Deploy OPNsense 22.7.5
2. Enable Squid transparent proxying
3. Try to browse the web
4. Many HTTP requests end with HTTP/409

Expected behavior

Proxy should just work, as it did before.

Describe alternatives you considered

No alternative possible except reverting to OPNsense 22.7.4.

Screenshots

None.

Relevant log files

Example of cache log entries:

2022-10-06T10:08:16     squid   kid1| SECURITY ALERT: on URL: self.events.data.microsoft.com:443
            current master transaction: master242934
2022-10-06T10:08:16     squid   kid1| SECURITY ALERT: Host header forgery detected on conn94738 local=52.168.117.170:443 remote=10.6.138.166:55762 FD 245 flags=33 (local IP does not match any domain IP)
            current master transaction: master242930
2022-10-06T10:08:16     squid   kid1| SECURITY ALERT: on URL: self.events.data.microsoft.com:443
            current master transaction: master242930
2022-10-06T10:08:16     squid   kid1| SECURITY ALERT: Host header forgery detected on conn94737 local=52.168.117.170:443 remote=10.6.138.166:55761 FD 245 flags=33 (local IP does not match any domain IP)
            current master transaction: master242906
2022-10-06T10:08:14     squid   kid1| SECURITY ALERT: on URL: amplify.outbrain.com:443
            current master transaction: master242906
2022-10-06T10:08:14     squid   kid1| SECURITY ALERT: Host header forgery detected on conn94707 local=23.57.82.102:443 remote=10.6.138.141:62888 FD 218 flags=33 (local IP does not match any domain IP)
            current master transaction: master242902
2022-10-06T10:08:14     squid   kid1| SECURITY ALERT: on URL: amplify.outbrain.com:443
            current master transaction: master242902
2022-10-06T10:08:14     squid   kid1| SECURITY ALERT: Host header forgery detected on conn94706 local=23.57.82.102:443 remote=10.6.138.141:62887 FD 218 flags=33 (local IP does not match any domain IP)
            current master transaction: master242886
2022-10-06T10:08:14     squid   kid1| SECURITY ALERT: on URL: cdns.gigya.com:443
            current master transaction: master242886
2022-10-06T10:08:14     squid   kid1| SECURITY ALERT: Host header forgery detected on conn94694 local=104.124.109.33:443 remote=10.6.138.141:62883 FD 205 flags=33 (local IP does not match any domain IP)
            current master transaction: master242882
2022-10-06T10:08:14     squid   kid1| SECURITY ALERT: on URL: cdns.gigya.com:443
            current master transaction: master242882
2022-10-06T10:08:14     squid   kid1| SECURITY ALERT: Host header forgery detected on conn94693 local=104.124.109.33:443 remote=10.6.138.141:62882 FD 205 flags=33 (local IP does not match any domain IP)
            current master transaction: master242874

Additional context

I regret that the to move to a new major version of Squid was pushed in an OPNsense maintenance release. Shouldn't this have been part of the next major release ?

Environment

OPNsense 22.7.5 (amd64, OpenSSL). KVM virtual machines Virtualized hardware

[fichtner]

I regret that the to move to a new major version of Squid was pushed in an OPNsense maintenance release. Shouldn't this have been part of the next major release ?

To be honest this comment falls flat on its face having held back on version 5 for stability reasons for a long time. Now that a security issue needed to be fixed the complaints come in? Seriously?

Cheers, Franco

[fichtner]

# opnsense-revert -r 22.7.4 squid

done...

[marinbernard-pep06]

I regret that the to move to a new major version of Squid was pushed in an OPNsense maintenance release. Shouldn't this have been part of the next major release ?

To be honest this comment falls flat on its face having held back on version 5 for stability reasons for a long time. Now that a security issue needed to be fixed the complaints come in? Seriously?

I'm sorry, no harm intended; I know nothing is simple. It's just that since OPNsense major releases require administrator validation (while minor ones do not), they seem like the ideal moment to include major changes: regressions are more likely to be noticed at that time because most admins will perform some kind of QA before triggering a mass-deploy.

[marinbernard-pep06]

# opnsense-revert -r 22.7.4 squid

done...

Yes, that's what I did, and locked the package too.

I checked the Squid Bugzilla. There exists a couple bug reports dealing with HTTP/409 and HHF. The most recent one is #4940, but there's been no activity since 2020. I think the issue is specific to environments which use Squid as an intercepting (transparent) proxy.

[marinbernard-pep06]

Hi again,

In fact, a similar issue was fixed with Squid 3 in February 2019 by including a patch from CentOS/NethServer to the ports tree (commit 546b753).

The patch has remained compatible with Squid 4 as the patched method (ClientRequestContext::hostHeaderIpVerify() in src/client_side_request.cc) did not change much between both releases. Squid 5, on the other hand, ships with important changes to this method: the context included in the original patch has vanished, and the patch probably fails to appy at build time.

So basically, I think we've reverted to Squid's unpatched behaviour (pre-2019).

Here's what the method looks like in Squid 4 and in Squid 5.

[marinbernard-pep06]

Applying the patch manually to the Squid 5 source tree results in patching the wrong method (clientFollowXForwardedForCheck) with fuzz=2. I don't now what max fuzz value is allowed when patching the ports tree at build time, but it it's > 0 I think this patch should be disabled until it is updated.

[fichtner]

Solved in ports for 22.7.6

[marinbernard-pep06]

Many thanks!