Curious: what is your usecase for custom ca-cert ?

[buxel]

Hi @suderman !

i reviewed the last few weeks of commits and wondered hwo you are using the custom CA. I thought it was going to replace letsencrypt but it does not seem to be the case (also: why should you do that?)

Could they be use for client certificate authentication? 🤔

I also noticed that when accessing immich from my phone, it showed an (seemingly invalid) certificate signed by the custom CA. Loading it on the browser, it showed the letsencrypt CA. Going back to the phone's browser and it also showed letsencrypt after a reload.

Enjoy your sunday!

[suderman]

It's not for client cert auth--I actually worked a project doing all that years ago, but today, hosting all my private services on a mesh network like Tailscale negates the need.

It's mostly cosmetic reasons, I suppose? With a private CA, I can create certificates for any hostname. This was kicked off by goal to reduce reliance on Internet availability when accessing local services on my home network.

Previously, I was dependent on Tailscale for the private 100.x IP addresses all my services listened on, Cloudflare for DNS records pointing to these IPs, and Let's Encrypt for HTTPS certificates for my Traefik reverse proxy. And all those solutions make perfect sense accessing publicly available services or private services while remote, but I wanted more resilience in a home scenario when the Internet has been disrupted.

To fix this, I've planned out the IP ranges for each of the networks I maintain servers: home, work and tailscale (VPS and mobile devices). I've read up on how to configure Tailscale routes and subnet routers so I can use my own DHCP 10.x IP ranges for home, work, etc giving me the best of both worlds--accessible via Tailscale but works locally without it. I've also deployed my own Blocky DNS service on each network to resolve custom hostnames so that will work locally as well. The last step is installing a CA root certificate on each non-NixOS device (my NixOS configs automatically include it) but that's a one-time step.

Most of the time, adding a root CA to the system store is enough, but there are exceptions. For example, Firefox has its own trust store (which I've accounted for in my config) and I think Python must have the same issue, hence the Immich complaint. Fortunately, the Immich app has a checkbox for to allow self-signed certs.

Circling back to the cosmetic reason, all my NixOS servers have short hostnames (lux, eve, etc), which is about the right length for a TLD. Now my hosted services have short URLs like https://immich.lux/ or https://jellyfin.eve/

I'll still use Let's Encrypt down the road when I setup a public URLs in a future configuration, but this made sense for me and my private services.

[buxel]

@suderman This ticks so many boxes for me! I was thinking about this idea some months ago but discarded it as i considered it too much hassle to manage. Looks like I should reconsider.

Two questions come to mind:

1. If you are at home, do you need tailscale on your client in order to access services on the VPS? For example home(phone) --> vps(gitea) From my understanding it should work without it, if blocky is taking care of the static routing.
2. A little more far-fetched: could the VPS additionally be used as a bastion host to expose services running at home? For example silverbullet.domain.com -> VPS -> TS -> home(lux) ? From my understanding the VPS to reverse proxy to silverbullet.lux in this example, right? I was toying with the idea to expose services like this for friends/family and not require them to use TS (or for myself using a device with limited privileges). Of course it would involve additional authentication on the VPS before letting them in.

I really appreciate your elaborate answers and that you are willing to write down all that knowledge. 👍

[suderman]

1. Nope. If you have a Tailscale subnet router running on your local LAN, and your LAN's router has static routes configured for Tailscale IP addresses, you're good to go. Any Tailscale-unaware devices on your WiFi will connect to your VPS via the subnet router's connection to Tailscale.

2. Yes, you can certainly expose private services through the magic of reverse proxies. In my configuration, the Traefik server is listening on ports 80 & 443 on all addresses (including public). However, all my private services include a IP whitelist middleware that looks like this:

  # Whitelist local network and VPN addresses
  services.traefik.dynamicConfigOptions.http.middlewares = {
    local.ipWhiteList.sourceRange = [ 
      "127.0.0.1/32"   # local host
      "192.168.0.0/16" # local network
      "10.0.0.0/8"     # local network
      "172.16.0.0/12"  # docker network
      "100.64.0.0/10"  # vpn network
    ];
  };

This rejects all public traffic. However, we could create an additional route on a separate public server that doesn't use this whitelist middleware. Instead we'll use a header middleware to get the hostname right, and also Let's Encrypt certificates. So long as the public server can talk to the private server via Tailscale, and you have a DNS record configured, this config will give your private origin server a public route:

  # Expose private service to public internet
  services.traefik.dynamicConfigOptions.http = let
    hostName = "notes.domain.com";
    origin = "silverbullet.lux";
  in {
    routers.silverbullet = {
      rule = "Host(`${hostName}`)";
      entrypoints = "websecure"; 
      tls.certresolver = "resolver-dns";
      tls.domains = [{
        main = "${hostName}"; 
        sans = "*.${hostName}"; 
      }];
      middlewares = "silverbullet";
      service = "silverbullet";
    };
    middlewares.silverbullet.headers.customRequestHeaders.Host = origin;
    services.silverbullet.loadBalancer.servers = [{ url = "https://${origin}:443"; }];
  };

Like you said, it might be a good idea to add an additional middleware for some kind of authentication.

[buxel]

For now, hosting everything private but not having tailscale required on all devices improves the WAF by a considerable amount. 👍

I initially struggled a little with iOS but it was just an additional switch missing in some buried settings.

[suderman]

Yeah, iOS does require several steps to trust a custom CA:

1. Download the crt file (iOS will call it a "configuration profile")
2. Settings > General > Downloaded Profile > Install
3. Settings > General > About > Certificate Trust Settings > Enable

For step one, the traefik module optionally serves the ca certificate on a port. So in my house, I can open Safari and enter the URL http://10.1.0.4:1234 and this will download and install the certificate on that device.

Even more bizarre is the Apple TV and accessing the hidden menu using the remote's play button:

1. Settings > General > Privacy & Security > Share Apple TV Analytics (hit the PLAY button) > Add Profile
2. Settings > General > About > Certificate Trust Settings > Enable