search

samcday / home-cluster

10 stars 0 forks source link

Caching Squid #491

Closed samcday closed 3 months ago

samcday commented 3 months ago

Should accept clients from anywhere in the cluster network, and from peers of the local home wired/wifi.

samcday commented 3 months ago

This was quite an adventure.

The ubuntu/squid Docker images helpfully do not build with OpenSSL by default. Instead Canonical have a squid-openssl package for that, but no published OCI images.

The bizarre thing is, whenever I try to use an openssl build of Squid in my k8s environment, Squid will chew up massive amounts of memory immediately on boot and get OOMKilled a couple of seconds later. I can reproduce this reliably by simply running kubectl run --rm -it alpine:3 foo -- apk add squid && squid --foreground -f /etc/squid/squid.conf.

Besides all that, though, the other issue with running Squid as cluster workload like this is that it excludes a few potential places from using it because of dependency issues.

So instead, I baked the Squid into the base image. It's configured with disk caching and SSL bumping. All nodes will run this Squid, and the idea is they all have cache_peer sibling config to resolve from each other's caches. All the necessary config is baked into the image, with the exception of the self-signed CA cert for SSL bumping (that's injected by Ignition).

samcday commented 3 months ago

The new Squid setup is working well.

It's rolled out across all nodes via the base image + ignition config. The cluster runs a pair of HAproxy pods to LB traffic into the squid peers. They all pull from each other's caches when resolving a MISS.

The node rpm-ostree is configured to proxy through squid. As it turns out my understand of the layering seems flawed, because it looks like all of the data is coming from OCI (I thought only the extra layers did, but it was somehow resolving the base stuff from the composes). Now that all that goes through harbor, we don't really need/want to cache that in Squid as well.

I also confirmed that it works well (enough) in Miniflux. I had some nice trauma flashbacks to a decade ago hacking/patching workloads to support http_proxy. Got those vibes because I needed to patch in the dodgy self-signed CA cert for the TLS mitm.

I've also hacked up some stuff to pull metrics + logs out of the squids.

I think I finally feel this is done enough to close out.