nixos/caddy: Caddy puts state in /var/lib/caddy/{.local/share/caddy,.config/caddy}

[lf-]

Describe the bug

Due to Caddy having wacky defaults for its configuration dirs (using xdg dirs for a system service), the service puts its state in hidden directories that are oddly structured.

https://caddyserver.com/docs/conventions#data-directory

The fix here is probably to set XDG_DATA_HOME to /var/lib/caddy. And, undocumentedly, probably XDG_CONFIG_HOME as well.

Steps To Reproduce

Steps to reproduce the behavior:

1. services.caddy.enable = true;
2. Run caddy.
3. ls -l /var/lib/caddy
4. ??? where is anything. it's hidden?

Expected behavior

Caddy should not store things in wacky hidden directories.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Notify maintainers

pkgs.caddy maintainers: @Br1ght0ne @emilylange @techknowlogick nixos module maintainers: not listed! another bug

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

• system: "aarch64-linux"
• host os: Linux 6.1.77, NixOS, 24.05 (Uakari), 24.05.20240207.f8e2ebd
• multi-user?: yes
• sandbox: yes

---

Add a :+1: reaction to issues you find important.

[mholt]

Thanks for the question/issue about the Caddy package for Nix; but... can I respectfully ask what is "wacky" about using XDG vars for resource paths?

We follow what is rather conventional for modern Unix systems, and the the Go standard library does essentially the same logic for choosing config, cache, and home folders.

??? where is anything. it's hidden?

It is not hidden, it is documented right on the page you linked to and in some of the logs Caddy emits, too. You can also discover paths using caddy environ.

[lf-]

Thanks for the question/issue about the Caddy package for Nix; but... can I respectfully ask what is "wacky" about using XDG vars for resource paths?

The wacky thing here, which I've never seen with any other service, is imitating the hidden (in a literal sense) directories one would see in a home directory, for a system service. This is mildly annoying for administration.

We follow what is rather conventional for modern Unix systems, and the the Go standard library does essentially the same logic for choosing config, cache, and home folders.

As I understand it, the XDG directories spec was not expected to be used on system services and thus its default subdirectories are unusual for system services.

??? where is anything. it's hidden?

It is not hidden, it is documented right on the page you linked to and in some of the logs Caddy emits, too. You can also discover paths using caddy environ.

By hidden I mean literally the paths have dots at the start of directory names.

[mholt]

the XDG directories spec was not expected to be used on system services

Huh, that's news to me. Is [this](the XDG directories spec was not expected to be used on system services) the right spec? Can you just show me where it says that it is not for system services?

What problems is the current code causing?

[francislavoie]

Worth mentioning that a change to the default at this point would cause user's data to get lost or improperly cleaned up if care isn't taken to move the data on upgrade. (I don't have a horse in this race, I'm not a Nix user, just mentioning that this is a sensitive change since Caddy's storage is the server's certs/keys.)

[lf-]

the XDG directories spec was not expected to be used on system services

Huh, that's news to me. Is [this](the XDG directories spec was not expected to be used on system services) the right spec? Can you just show me where it says that it is not for system services?

What problems is the current code causing?

The primary issue is confusion due to unexpected hidden directories, which aren't at all typical practice for system services.

The base directory spec is full of references to "user specific state" or similar kinds of things that strongly imply it's not intended to be used for system services. If you read the systemd.exec man page under StateDirectory, there's a table of used directories and the xdg spec is only used for user services there.

It's not a problem(tm) that the layout is this way but it will confuse every single user who is looking for the certs (which I'm also unsure if the layout has any stability guarantee in regards to that) at least once, before they figure it out. We could migrate to something different on new values of system.stateVersion, for example.

Concretely we would probably set XDG_DATA_HOME deliberately such that it would be in /var/lib/caddy/data instead of /var/lib/caddy/.local/share/caddy, and likewise for XDG_CONFIG_HOME.

[emilylange]

I would like to clarify that migrations like these are not an issue in nixpkgs/NixOS and can be handled appropriately without data loss :)

[kanashimia]

Setting XDG_DATA_HOME=/var/lib/caddy/data means that the data would be in /var/lib/caddy/data/caddy which is pretty weird too. I'm not really sure if this is a good idea, but we could do like this:

systemd.services.caddy = {
  environment = {
    XDG_DATA_HOME = "/var/lib";
    XDG_CONFIG_HOME = "/var/lib";
  };
};

Then data will be stored directly in the /var/lib/caddy. This would imply removal of the ability to override the state dir location, I don't think that's much of a loss. XDG_CONFIG_HOME can be removed if we somehow set persist_config off.

Would be nice if caddy added some other way to set state directory, some special env variable that takes precedence over the XDG directories would be good, like the one in caddy v1.

According to the man systemd.exec systemd sets $STATE_DIRECTORY env variable automatically depending on the StateDirectory service entry, caddy can add support for it, preferring it over XDG dirs, but it means that they will have to do migrations on their side, has benefit as the other distros will have the same location too then. It will make using caddy marginally more difficult in another service I would imagine, like if some other app that stores it's data in the /var/lib launches caddy, does anyone do that?

Also note that caddy is not the only service that creates dot dirs like that, here are some examples from my own server:

~ 
! l /var/lib/jibri 
total 44K
drwxr-xr-x 3 993 990 4.0K Feb 18  2022 .cache
drwx------ 4 993 990 4.0K Feb 18  2022 .config
drwx------ 2 993 990 4.0K Feb 18  2022 .icewm
drwx------ 3 993 990 4.0K Feb 18  2022 .pki
-rw-r--r-- 1 993 990  552 Feb 18  2022 .asoundrc
-rw-r--r-- 1 993 990   16 Feb 18  2022 icewm.preferences
-rw-r--r-- 1 993 990   10 Feb 18  2022 jibri.conf
-rw-r--r-- 1 993 990 1.7K Feb 18  2022 logging.properties
-rw-r--r-- 1 993 990  490 Feb 18  2022 pjsua.config
-rw-r--r-- 1 993 990 6.7K Feb 18  2022 xorg-video-dummy.conf

~ 
! l /var/lib/acme 
total 12K
drwxr-xr-x 4 acme acme 4.0K Aug 31  2021 .lego
drwxr-x--- 2 acme acme 4.0K Aug 21  2021 .minica
drwxr-x--- 2 acme acme 4.0K Mar 28 16:24 redpilled.dev

~ 
! l /var/lib/terraria 
total 8.0K
drwxr-xr-x 3 terraria terraria 4.0K Jan 20 14:06 .local
drwxr-xr-x 2 terraria terraria 4.0K Apr 26 15:44 worlds

That's just a minority of services though, and that is not the correct convention. Actual convention is described in the man file-hierarchy or man hier, you can read it here: https://man7.org/linux/man-pages/man7/file-hierarchy.7.html You can clearly see it being split into "user packages" and "system packages".

For NixOS we would need to support both options at the same time, choosing which one to use depending on the system.stateVersion, and will optionally need to provide migrations, sometime later we can remove that logic.

So overall this is not exactly a trivial change for something so trivial.

[kanashimia]

I found that you can actually configure data dir location from the config: https://caddyserver.com/docs/caddyfile/options#storage So you can do for example storage file_system {$STATE_DIRECTORY} or storage file_system ${config.services.caddy.dataDir} Although it still creates /var/lib/caddy/.local/share/caddy/instance.uuid and /var/lib/caddy/.config/caddy/autosave.json for some dubious reason. Has downside that it would require user to configure it too if he specifies services.caddy.configFile

[francislavoie]

Although it still creates /var/lib/caddy/.local/share/caddy/instance.uuid and /var/lib/caddy/.config/caddy/autosave.json for some dubious reason.

The former is because this is created before the config is loaded, because it stores an random instance ID (created once, reused there-on) for tracking which Caddy instance is doing what in certain scenarios where Caddy is clustered. It's also important that it doesn't use the same networked storage if one is configured (e.g. Redis storage backend) because then all Caddy instances connected to Redis would be using the same instance ID, defeating the purpose of that. So it always uses the filesystem.

The latter (autosave.json) is stored in the config storage directory, which does not follow data storage config. See https://caddyserver.com/docs/conventions#file-locations which explains the difference. This file is Caddy's last config; this isn't particularly useful if you use a Caddyfile, but if you use Caddy's API for making config changes, it's vital to ensure you don't lose you config changes performed via the API, when Caddy restarts. You can turn this off with https://caddyserver.com/docs/caddyfile/options#persist-config if you want.

[mholt]

The former is because this is created before the config is loaded

To clarify, the instance ID is loaded from the file system, regardless of configured storage. Caddy does not load the instance ID from the configured storage.