[Feature][Proxy] Add support for Windows updates and Debian package caching

[borisneubert]

With the right configuration, Squid is able to cache Windows updates [1] and Debian packages [2]. This requires to add custom refresh_pattern directives to the Squid configuration file which is currently not possible via the web interface. Manual edits to the Squid configuration files do not survive a reboot.

Maybe it is sufficient to include a free text field with configuration options that get added to the configuration file.

Making the Squid cache persistent over a reboot would be a plus - as far as I have observed a reboot empties the Squid cache /var/squid/cache.

[1] http://wiki.squid-cache.org/SquidFaq/WindowsUpdate [2] https://www.midnightfreddie.com/using-squid-to-cache-apt-updates-for-debian-and-ubuntu.html

[AdSchellevis]

@borisneubert you can easily add custom hooks (which survive reboots) using pre-auth and post-auth config locations, defined here https://github.com/opnsense/core/blob/17.1.8/src/opnsense/service/templates/OPNsense/Proxy/squid.conf#L273 and https://github.com/opnsense/core/blob/17.1.8/src/opnsense/service/templates/OPNsense/Proxy/squid.conf#L283 .

It could be an idea to see what we miss in terms of features and what's needed to configure both options, but we won't add plain text fields to input configuration data directly. For both security and validation reasons.

[fichtner]

I agree, we should take those two up as checkboxes, they come up quite frequently

[borisneubert]

Thank you. Added config files with refresh patterns to the post-auth folder and acls for manager access from the local network to the pre-auth folder. squid -k parse shows that this is fine. Was I supposed to find this in the documentation?

Could you please help me understand the security and validation issues the plain text fields would pose? In my view, the only difference between adding own squid directives by file instead via the UI is that the former creates a hurdle while the latter requires at least some sophistication on the end of the firewall admin.

In any case, making the option for custom hooks visible in the UI would be a plus.

[AdSchellevis]

The documentation needs work on this subject, we just can't do all at once.

Raw config data can't be validated, which could lead to broken configurations, that's one of the main reasons why we don't support plain text fields to inject config data. We try to keep things user friendly and as consistent as possible.

When possible we keep the option open for advanced users using plugins and hooks like the one described here (it's not very hard to copy a file to the firewall).

If you share your configuration, we could look into adding the feature using a simple checkbox, which would benefit more users.

[borisneubert]

Thanks for the explanation. I wanted to share some documentation anyway after thorough testing.

I am yet unsure whether my config for caching debian packages and windows updates actually works. This needs some investigation after the next patchday and having a close look into the cache. The post-auth hooks currently look like this:

package-cache.conf

refresh_pattern deb$           0       20%     4320 refresh-ims
refresh_pattern udeb$          0       20%     4320 refresh-ims
refresh_pattern Packages\.bz2$ 0       20%     4320 refresh-ims
refresh_pattern Sources\.bz2$  0       20%     4320 refresh-ims
refresh_pattern Release\.gpg$  0       20%     4320 refresh-ims
refresh_pattern Release$       0       20%     4320 refresh-ims

update-cache.conf

# http://wiki.squid-cache.org/SquidFaq/WindowsUpdate
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 129600 reload-into-ims
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 129600 reload-into-ims
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 129600 reload-into-ims

The pre-auth hook for accessing the cache manager from LAN works. It looks like follows:

manager.conf

# http://wiki.squid-cache.org/Features/CacheManager
acl managerAdmin src 192.168.31.0/24
http_access allow managerAdmin manager

The latter is well-suited to be exposed through the web interface (just an input field for an IP address or network). But that might be too special a case for consideration.

[borisneubert]

Update: after some weeks I can confirm that windows updates end up in the cache and stay in the cache given the aforementioned update-cache.conf

[AdSchellevis]

@borisneubert thanks for reporting back, as soon as I can find some time I will see if I can add the checkboxes.

[elico]

I am not sure if it's the right place or not but I wrote a windows update caching proxy which works with squid as a cache_peer: http://www1.ngtech.co.il/wpe/?page_id=301

It runs on systems with 10Gbps+ so... pretty nice. Is it helping?

[borisneubert]

@elico That would make a nice Update Accelerator plugin like they have in IPFire which I used before turning to OPNSense.

[fabianfrz]

@AdSchellevis arch linux package have the suffix .*\.pkg\.tar\.xz and red hat (like) systems use .*\.d?rpm (d means delta update). They should be appended to the list of @borisneubert. Probably use the same settings as for the debian packages.

[fabianfrz]

@AdSchellevis I'll add it

[elico]

@AdSchellevis You might also want to use a StoreID helper to de-duplicate different servers for updates. Anyone have written a helper for this?

[elico]

@borisneubert Well I will need to understand how it can be integrated on FreeBSD since it needs to have a cron job running every now and then(5 minutes) and also a clean-up every hour for old cached files. Something like "ls /path && find /path -t f 'older then x days'|xargs rm".

[borisneubert]

@elico For reference the links to IPFire's (FreeBSD) update accelerator documentation here and here and the source. IPFire removes outdated files, files without source and old files (all optional, based on configuration).

[AdSchellevis]

@borisneubert @elico can you test the patches contributed by @fabianfrz https://github.com/opnsense/core/commit/8334db3a7b0478d473984b594a64b44513bf57a2 ?

[borisneubert]

@AdSchellevis This means removing my pre-auth and post-auth hooks and running

opnsense-patch -c core 8334db3a7b0478d473984b594a64b44513bf57a2

Right?

[AdSchellevis]

@borisneubert yes that should do the trick

[borisneubert]

@AdSchellevis patch did the jobs - have removed the manually entered patterns and checked the boxes.

squid -k parse shows

2017/10/29 17:52:45| Processing: refresh_pattern pkg\.tar\.xz$   0       20%     4320 refresh-ims
2017/10/29 17:52:45| Processing: refresh_pattern d?rpm$          0       20%     4320 refresh-ims
2017/10/29 17:52:45| Processing: refresh_pattern deb$            0       20%     4320 refresh-ims
2017/10/29 17:52:45| Processing: refresh_pattern udeb$           0       20%     4320 refresh-ims
2017/10/29 17:52:45| Processing: refresh_pattern Packages\.bz2$  0       20%     4320 refresh-ims
2017/10/29 17:52:45| Processing: refresh_pattern Sources\.bz2$   0       20%     4320 refresh-ims
2017/10/29 17:52:45| Processing: refresh_pattern Release\.gpg$   0       20%     4320 refresh-ims
2017/10/29 17:52:45| Processing: refresh_pattern Release$        0       20%     4320 refresh-ims
2017/10/29 17:52:45| Processing: refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip)     4320 80% 129600 reload-into-ims
2017/10/29 17:52:45| Processing: refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 129600 reload-into-ims
2017/10/29 17:52:45| Processing: refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip)       4320 80% 129600 reload-into-ims
2017/10/29 17:52:45| Processing: refresh_pattern ^ftp:          1440    20%     10080
2017/10/29 17:52:45| Processing: refresh_pattern ^gopher:       1440    0%      1440
2017/10/29 17:52:45| Processing: refresh_pattern -i (/cgi-bin/|\?) 0    0%      0
2017/10/29 17:52:45| Processing: refresh_pattern .              0       20%     4320

Squid starts fine. Will observe caching behavior.

[elico]

@borisneubert thanks for the details despite to the fact that it's an OpnSense issue.

[AdSchellevis]

@borisneubert thanks for confirming, I'll close this issue then.

[FathiBenNasr]

Hi, Is there any sample full squid config for such a purpose as caching windows updates, maybe with the above snippets updated for windows 10 ?