Any good way to handle `/etc/localtime`? (Or: "How to persist symlinks as symlinks?")

alex-robbins commented 10 months ago

/etc/localtime is a file with unusual semantics. It must be a symlink that points into /etc/zoneinfo at a timezone file. The name of the timezone is then parsed (!) from the symlink target. This is explained in the manpage (note that the zoneinfo directory is under /usr for upstream systemd).

(In all fairness, I think this is a systemd bug: it's ridiculous to use a symlink this way. localtime should just be a normal file whose contents is the name of a timezone, and systemd should then find that file in the zoneinfo directory. But, unless someone wants to make this argument to the systemd people, this is the interface we're stuck with.)

The problem for impermanence is that it doesn't seem to provide any way to create a file like this. For example, say you want to persist the timezone at /persist/etc/localtime, so you set environment.persistence."/persist".files = [ "/etc/localtime" ]. Then:

If /persist/etc/localtime doesn't exist or is a symlink to a file that doesn't exist (e.g. its target is zoneinfo/America/Los_Angeles),
- then you get an /etc/localtime that is a symlink to /persist/etc/localtime,
- which obviously does not work, because although it's a symlink, it doesn't directly (or even indirectly) point to a file in /etc/zoneinfo (it's a broken link, or a link to a broken link, if you prefer)

Alternatively:

If /persist/etc/localtime is a symlink to a file that does exist (e.g. its target is /etc/zoneinfo/America/Los_Angeles),
- then impermanence creates a bind mount, and you get an /etc/localtime that is (a bind mount of) a normal file,
- and this also doesn't work, because although the file exists and contains valid TZ data, it still isn't a symlink to a file in /etc/zoneinfo—it's a normal file, not a symlink

It's also worth mentioning that even if you go outside of impermanence and manually set things up such that /etc/localtime is a symlink to /persist/etc/localtime, which in turn is a symlink to /etc/zoneinfo/America/Los_Angeles, this still doesn't work, because /etc/localtime doesn't directly point to a file in /etc/zoneinfo and the TZ name cannot be parsed from it.

Also, to be clear, when I say "does not work" in all of the above cases, I mean that systemd ignores /etc/localtime and the timezone ends up being UTC.

When is this issue visible?

I'm guessing that most people using /etc on tmpfs don't see this issue because they set time.timeZone to a non-null value in their nixos configuration. However, this disables imperative TZ configuration with timedatectl, so it may not be appropriate for all situations (e.g. a laptop, where you don't want to rebuild the system configuration when changing timezones).

Solutions?

It sounds like Linux 5.12+ might have support for bind mounting symlinks as symlinks, but this capability doesn't seem to be exposed by the usual userspace tools, and I couldn't really find any information about it outside of that StackExchange answer, so I don't even know how it works.

Workarounds

The best workaround I can think of is to make a systemd service that copies the symlink verbatim out of /persist when it starts and copies it back when it stops. Of course, this wouldn't persist changes if the system loses power, etc; that would take a daemon watching for changes. Ugh.

ckiee commented 9 months ago

I ran into this here too. Exactly the same case with a laptop that travels between timezones every year or two.

(In all fairness, I think this is a systemd bug: it's ridiculous to use a symlink this way. localtime should just be a normal file whose contents is the name of a timezone, and systemd should then find that file in the zoneinfo directory. But, unless someone wants to make this argument to the systemd people, this is the interface we're stuck with.)

poettering doesn't seem too interested in fixing this: https://github.com/systemd/systemd/issues/22514

mboyea commented 6 months ago

Relevant Problem: If I persist the entire /home/<user> directory, it copies symlinks within it correctly. However, if I try to persist /home/<user>/.nix-profile (a symlink) directly, it copies incorrectly. This means it seems I can't have a local installation of home-manager persist between rebuilds with impermanence, unless I persist the entire user directory.

Solution: Since we can't directly persist symlinks, but we can persist directories with symlinks inside them, we want to move the symlink to a subdirectory. Change the directory of the nix user profile to comply with the XDG Base Directory Spec by adding the following to /etc/nixos/configuration.nix:

nix.settings.use-xdg-base-directories = true;

Then tell persistence to persist the following (assuming $XDG_STATE_HOME is not set, nix defaults to .local):

users.<user> = {
  ".config/nix"
  ".local/state/nix"
  ".local/share/nix"
  ".config/home-manager"
  ".local/state/home-manager"
  ".local/share/home-manager"
}

Now, after sudo nixos-rebuild boot & reboot & home-manager switch, user home-manager packages persist between reboots. My user-defined home-manager setup can persist, and I still maintain opt-in persistence!

References: https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-profile#user-profile-link https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-use-xdg-base-directories

nix-community / impermanence