caddy: add all addons

[davidak]

compile caddy with all addons listed on the download page.

Only middleware handlers which are invoked from the Caddyfile will be chained in, so small Caddyfiles are very fast and efficient.

source: Directive section on https://caddyserver.com/docs/caddyfile

a later improvement could be to select just the addons you want like vim_configurable.

@angus-g

[ntninja]

FYI, if you just came here looking for a way to build Caddy with extra modules:

Just use the Nix expression from PR https://github.com/NixOS/nixpkgs/pull/259275, it actually works and you can just copy-paste it (don’t try to get the PoC posted here to work like it did – the version from the PR is much better and easier to use!): https://github.com/NixOS/nixpkgs/blob/a33b02fa9d664f31dadc8a874eb1a5dbaa9f4ecf/pkgs/servers/caddy/default.nix

[winterqt]

The main issue with the above approach (as well as the one added to CoreDNS) that the above PoC doesn't have is that users only need to specify the hash of the plugins they want to add, not the entire tree. Sadly, it doesn't seem like such an approach is actually scalable.

[MROvaiz]

how do i use plugin? do i have to compile anything? any doc?

[ntninja]

@MROvaiz If you’ve never used overlays before it goes like this:

1. Save the contents of https://raw.githubusercontent.com/jpds/nixpkgs/caddy-external-plugins/pkgs/servers/caddy/default.nix as /etc/nixos/overlay/caddy-custom/default.nix
2. Create an overlay function in file /etc/nixos/overlay/default.nix (this can be extended for adding other external packages later if you want):

   
   final: prev:

{

Updated Caddy with support for including extra modules

#

Source: https://github.com/NixOS/nixpkgs/pull/259275

caddy-custom = prev.callPackage ./caddy-custom {}; }

3. Reference the overlay function from `/etc/configuration.nix`:
```nix
{
   # … other settings …

   # System package customization
   nixpkgs.overlays = [
      (import ./overlay)
   ];

   # … other settings …
}

4. Override the Caddy derivation used by the Caddy service in whatever file you’re setting services.caddy (can also be in /etc/configuration.nix):

   {
   services.caddy = {
     package = pkgs.caddy-custom.override {
        externalPlugins = [
           {name = "transform-encoder"; repo = "github.com/caddyserver/transform-encoder"; version = "58ebafa572d531b301fdbc6e2fd139766bac7e8d";}
           {name = "connegmatcher";     repo = "github.com/mpilhlt/caddy-conneg";          version = "v0.1.4";}
        ] ++ (
           # Caddy Layer4 modules
           lib.lists.map (name: {
              inherit name;
              repo = "github.com/mholt/caddy-l4";
              version = "f3a880d4c01c884f4a096ccceb6c6d1e2d1d983d";
           }) ["layer4" "modules/l4proxy" "modules/l4tls" "modules/l4proxyprotocol"]
        );
        vendorHash = "sha256-7cRI65foALEsfYhvdGresq7oma/cIsnVtbq+Gan5DCU=";
     };
   
     # … other caddy settings …
   };
   }

Hope this helps!

[MROvaiz]

@ntninja Thank you, it's working. so I'm using yours overlay package build derivation. so if there is an update in caddy, what do i have to do? or i have to wait for this derivation be merge with nixpkg git?

[airone01]

PR @ #317881

[crabdancing]

@ntninja I've tried this, but it's not working for me. I recreated the problem in a flake for easier debugging:

https://github.com/crabdancing/nixos-caddy-with-plugins

It says:

warning: creating lock file '/home/ada/projects/nixos-caddy-with-plugins/flake.lock'
error: builder for '/nix/store/apjbwk230vf97ckwcl8rqa7dd7lqln3q-caddy-2.7.5.drv' failed with exit code 1;
       last 10 log lines:
       > Running phase: unpackPhase
       > unpacking source archive /nix/store/6qhv0b3ripxgjyw8jhibqmn1i3r91bzs-source
       > source root is source
       > Running phase: patchPhase
       > Running phase: updateAutotoolsGnuConfigScriptsPhase
       > Running phase: configurePhase
       > Running phase: buildPhase
       > Building subPackage ./cmd/caddy
       > # github.com/caddyserver/caddy/v2/modules/caddytls
       > modules/caddytls/automation.go:254:18: cannot use func(name string) error {…} (value of type func(name string) error) as func(ctx context.Context, name string) error value in struct literal
       For full logs, run 'nix log /nix/store/apjbwk230vf97ckwcl8rqa7dd7lqln3q-caddy-2.7.5.drv'.

Does anyone have any ideas?

[airone01]

@crabdancing

• Use https://raw.githubusercontent.com/Nanotwerp/nixpkgs/caddy-rebase/pkgs/by-name/ca/caddy/package.nix instead of the one from the old PR for package.nix
• Rename externalPlugins to caddyModules in configuration.nix and remove vendorHash
• In package.nix, remove the argument for caddyModules from withPlugins, and move it to the top function arguments as caddyModules ? []

Now it should work. Again this is a hack.

Edit: this does not actually work if you use it in an overlay, but crabdancing made a nice flake (see comment below) which I'm using in my setup. If you are coming from Google, Ignore my response above and head over to her flake, or wait for the PR to be merged 👍

[crabdancing]

@airone01 Thanks so much! I'm very happy with my packaging for this now. :) It may be a hack, but it's the least hacky hack that we have for now.

For anyone struggling with using caddy-l4 or other Caddy modules / Caddy plugins declaratively on NixOS, I wrote a flake you can use for it here. I hope it can help someone :)

[ntninja]

@ntninja Thank you, it's working. so I'm using yours overlay package build derivation. so if there is an update in caddy, what do i have to do? or i have to wait for this derivation be merge with nixpkg git?

1. Open /etc/nixos/overlay/caddy-custom/default.nix and change the version number and make the value of hash inside dist and src empty
2. Next set the vendorHash value inside the services.caddy configuration also to empty
3. Run nixos-rebuild switch --fast (it should print “warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA='” exactly three times!)
4. The switch will fail but should print you the correct value for the hash field inside src as ”actual” value – copy that value
5. Edit /etc/nixos/overlay/caddy-custom/default.nix again and change the hash value inside src to the copied value
6. Run nixos-rebuild switch --fast again and repeat steps 3 and 4 for the hash value inside dist
7. Run nixos-rebuild switch --fast again and use the “actual” value as vendorHash inside services.caddy
8. Run nixos-rebuild switch --fast one last time and hopefully this time it should work

@crabdancing It’s probably that some module you’re using isn’t compatible with the older version of Caddy that was used in my guide, try repeating after doing the steps above for Caddy 2.8.4! (Also your profile text speaks to me. :wolf:)

[ntninja]

(Also, I’ll try doing a write-up on how I think plugin support could be done in a NixPkgs-acceptable way later on. Just know I probably won’t have time to implement it myself in any reasonable timeframe.)

[Diti]

It would be neat if we could use caddyPackages in nixpkgs the same way we use nodePackages, python3Packages or vimPlugins!

[airone01]

It would be neat if we could use caddyPackages in nixpkgs the same way we use nodePackages, python3Packages or vimPlugins!

Is it possible? Considering the modules are used at compile time I mean. I'm a newbie so correct me if I'm wrong. Also if it is indeed possible, I'm down to becoming a maintainer for some plugins.

[ntninja]

@airone01 Yes, it should be possible. Unfortunately, it's nontrivial to even explain the problem and how it can be solved (very in-depth Nix). But since there is obviously interest, I’ll try sketching it out soon (can’t right now, we don’t have much time per day, sorry) – someone else will need to fumble with Nix derivations to actually make it work though.

[ntninja]

Background on derivations

Before trying to fix this it is important to understand the two major kinds of derivations in Nix: In general derivations take some kind of input and run some command (typically a shell script) that produces some desired output.

If you’ve ever written a derivation before, it was most-likely a so-called fixed-input derivation: You can specify/use some arbitrary inputs and some (shell) command(s); once all the required inputs are available the Nix program will then run the specified command(s) are run it in an isolated build environment with access only to the used inputs (and, in particular no network access!) – the so-called “sandbox” you might have seen mentioned. The idea here is for fixed-input derivations we expect the command(s) basically just be some arbitrary-complex transformation from the given inputs to whatever output the command(s) result in. Notably we do not need to specify what the output actually will be since we know (within some minor margin of error) that it will essentially just be some alternate representation of the input.

The other major type of derivation is the so-called fixed-output derivation: Again you specify/use some arbitrary inputs and some (shell) command(s), but this time you also specify the hash of the resulting output. Unlike fixed-input derivations, fixed-output derivation are not run in a fully isolated environment and can access the network – there is no need to limit their inputs after all since we know there is only one possible output in may produce! Notably the fetchFromGitHub function produces a fixed-output derivation that downloads an archive from some GitHub URL from the internet; that means it may return essentially arbitrary data, however since you also need to specify the expected hash we know GitHub cannot lie to us and sometimes return something else.

The currently suggested approach for Caddy plugins

If you look at the current suggestion for external modules in Caddy, you can see that the externalPlugins attributes is used in only two places: modBuildPhase and preBuild; both of these simply patch the Caddy main entry point in a special upstream-supported way to also reference any plugins you specify. (This is also what the xcaddy tool does internally.)

However this brings us to the actual problem in terms of NixPkgs packaging: All we did is add some references to these external plugins, we have not actually made their source code available locally at all! Since building Caddy itself uses a fixed-input derivation (see above) we’d now get errors as the Go tool attempts to download these dependencies from the internet – something a fixed-input derivation is not allowed to do.

The reason why the approach still kinda works is because we haven’t just patched the preBuild phase (which would result in the error mentioned above), but also the modBuildPhase: The modBuildPhase is fixed-output derivation (run before the fixed-input build phase) that uses the go mod vendor tool to download all the needed sources of all project dependencies and store them as derivation output – after patching the Caddy source code its list of dependencies includes both all the dependencies of Caddy itself, as well as all the dependencies of all the plugins it now uses!

There is a major inconvenience with this however: Since it’s a fixed-output derivation we had to specify the expected output hash of this step and this hash, the vendorHash, now covers both Caddy itself as well as all its added dependencies – and since users of the externalPlugins need to also specify an updated combined vendorHash this means that every time there is a minor update of Caddy in NixPkgs users’ systems will start encountering errors when upgrading!

A potential solution

Basically, what we need to do is treat each Caddy plugin as its own separate Go project that we run go mod vendor for: One fixed-output derivation for Caddy itself (as is currently in NixPkgs), then one extra fixed-output derivation using go mod vendor for each Caddy plugin. This means that for each plugin we also need to specify its own vendorHash and source, in addition to the Go module path that is currently specified. After downloading each vendor directory separately as fixed-output derivations, we then need to then merge them using a fixed-input derivation and use that as Go vendor directory when building Caddy – I’m not aware of any official way to merge Go vendor directories, but reading https://go.dev/src/README.vendor it appears that just running cp -r --reflink=auto "$PLUGIN_VENDOR_DIR" "$CADDY_VENDOR_DIR" for every plugin may be enough to “merge” them.

It should be possible to do that last step as part of configurePhase of the main Caddy build even, as long as the result ends up in a directory called vendor inside the Caddy sources being built.

So in to total it would be something like this:

1. For each plugin:
   1. Download its sources (using a fetch* function)
   2. Run go mod vendor and capture the result in a fixed-output derivation (write it to $out)
2. When building Caddy:
   1. Merge all plugin vendor directories into vendor/ during postConfigure
   2. Keep the part about patching Caddy in preBuild

… maybe, there are probably several catches and pitfalls. You’ll need to experiment to actually get it to work. (You can also look at the buildGoModule source code for inspiration: https://github.com/hsjobeki/nixpkgs/blob/ad513b9c4fdbdc2f48faafdc802817536cb9c9b6/pkgs/build-support/go/module.nix#L3:C1)

Good luck!

(@airone01 why :smile: ?)

[airone01]

@ntninja I thought it was a "happy" emoji but it's a "laugh" emoji. My bad

Also great explanation, it's really detailed and simple and really helps a Nix newbie like me to understand how it works behind the scene. Thanks for that! I won't have time to code this month because of an entrance exam, but I'll get right back at it after.

[W1M0R]

@ntninja Great explanation! Thanks, I've written many derivations before, but now it all makes sense to me.

[diamondburned]

@ntninja I think gomod2nix already does quite a lot of this, especially the combining dependency derivations into a single usable vendor directory. This could be a good reference as well.

[ntninja]

@airone01 Ah, I see :grin:

@diamondburned Yes, it does. I just looked it over and it sadly cannot be used from inside NixPkgs directly: You’d have to run it ahead-of-time for every Caddy release (can be done on every Caddy update using a PR), as well as for every plugin you’d like to support (meaning every plugin needs to be also packaged in NixPkgs ahead-of-time, since users would be required to locally run that tool each time they add or update one that is not pregenerated in NixPkgs)*. In addition that tool does not appear to actually support merging dependency derivations, that code would still need to be written regardless.

So based on this – while it certainly is a nicer solution than the current giant-vendor-directory approach – it seems the tool is mostly orthogonal to the problems the Caddy packaging has.

* The blog post has a background on why – technically this could also be solved using import from derivation, but that’s forbidden for NixPkgs.

[escherlies]

Using a nix flake to build caddy with modules

The solution works great for us, so i wanted to share it here:

NixOS Flake for Caddy With modules

The whole process is outlined in the readme.

Big shoutout to Pinpox 😊