Darwin installer: create a case sensitive APFS volume

[roberth]

Platform

• [ ] Linux:
• [x] macOS
• [ ] WSL

Additional information

We have the "case hack" to make all needed Nix file system objects representable on a case-insensitive file system, like macOS currently has for the store, but that doesn't mean that those files are easily usable; quite the contrary.

Users run into problems e.g. when doing things like building a linux image on darwin. (example)

Ideally we don't need the case hack, and indeed since we're already creating a volume, we might as well make it case sensitive.

This doesn't necessarily mean removing the case hack feature altogether. It's a useful feature for use with all case insensitive file systems; perhaps also Windows. For instance storing a cache on a case insensitive fs should work fine with it. (Though not sure it would today, because our data model is not up to scratch, https://github.com/NixOS/nix/issues/9318)

Plan

• Make Nix warn when building using a case-insensitive store
• Make Nix warn when building using a case-insensitive build-dir
• Provide documentation for making /nix case sensitive and link to it from the warnings
• Change the installer to make a case sensitive /nix
• Change the installer to make a case sensitive build-dir (wherever that is)
• Make sure TMPDIR in the build environment also points to a case sensitive volume

Priorities

Add :+1: to issues you find important.

[abathur]

Since I'm floating some ~challenges, I'll start by saying that I would like to see this happen myself, and took explicit action in the installer to make this easier for people to fiddle with so that the adventurous among us could start discovering the downsides.

I'm not one of these people, but the few things I'm aware of are:

• The utility of a case-sensitive store may be constrained if builds are still done in /tmp on the default-insensitive root volume. (Maybe this work can be moved under /nix, but then we don't get to lean on the OS cleaning it up on reboot and probably need to actively manage it? It sounds like your notes above mean something along these lines, but I don't see clean-up addressed.)
• @reckenrode has noted (and AFAICT has upstreamed fixes for) some meson issues on a case-sensitive store. Hopefully he can elaborate as to whether that's an ongoing or solved problem.
• ~If we start migrating the install base, over time we'll lose eyes on namespace clashes that cause trouble here, but there may still be some meaningful fraction of the install base with old case-insensitive installs. (This is probably just a thing to keep in mind, but maybe Nix could also be doing some explicit detection of these and erroring/warning on them during a transition period before we finally shrug and suggest people with insensitive stores uninstall/reinstall?)~ Sounds like you are just thinking we use the warnings to nudge people in this boat to fix up. That should be fine.
• This one's speculative/theoretical. Some people work around problems we have getting nix-installed Mac apps in a number of ways, some of which include copying the .app files out of the store and into an Applications directory (because symlinks cause trouble). I wonder a little bit if we'll see any weird behavior plop out if the apps (which probably aren't rigorously tested on case-sensitive file systems) build on a case-sensitive FS and then get copied out to an insensitive one.

I'll mention this in the macOS room, but I would also recommend searching the macOS room for case-sensitive and clicking through the discussions around this, since it's been discussed at least a few times over the past few years.

[abathur]

Sorry--I'm a little scattered from storm-related displacement in Houston and did a poor job of absorbing implications of bullets in OP. Edited a little to improve but probably still rehashes stuff y'all already have in mind.

One more, though:

Provide documentation for making /nix case sensitive and link to it from the warnings

When we've seen this come up in the macOS room, I think it was reported (maybe by @lilyball?) that there isn't a way to convert the volume. On the up side, this probably means that the "instructions" can just be 2 steps linking to uninstall and reinstall instructions?

[reckenrode]

The next time I upgrade my system, I probably won’t use a case-sensitive store unless I’m given no choice because it doesn’t solve case-sensitivity problems while building (because /tmp is case-insensitive) while adding extra hassles.

I’ve experienced two sets of issues with using it:

• Meson occasionally has problems with case-sensitivity. See:
  • https://github.com/mesonbuild/meson/pull/11820
  • https://github.com/mesonbuild/meson/pull/12587
  • https://github.com/mesonbuild/meson/pull/13038
• gnatcoll-python3 fails to build even though the casing of the file is actually correct (see https://github.com/NixOS/nixpkgs/pull/310450#issuecomment-2103622184). I have no idea why. 🫤

If the store is made case-sensitive by default, then I hope the builders are converted immediately. Otherwise, that will hide compatibility issues that will cause problems for users when they try or need to build things from source.

[abathur]

Lightly edited reproduction of further Matrix discussion on the tmpfile cleanup angle:

Randy Eckenrode

I missed the last bullet. I don’t think warning about TMPDIR is enough. Nix needs to pick a new location because TMPDIR will always be case-insensitive on Darwin. It also needs to make sure that location gets cleaned
regularly because /tmp accumulates a ton of build cruft.

Huh. When did macOS drop daily tmpfile cleaning? /etc/periodic/daily/110.clean-tmps is present on my wife’s iMac (running macOS 13.whatever) but missing on my > MBP (running macOS 14.5). macOS 14. I checked another Mac that was upgraded from 13.x, and the file is also gone.

For cleanup, couldn’t nix just install its own clean-tmps for the nix build location?

pxc@pxc:matrix.org

does it no longer clean those files, or just rely on a different mechanism for it?

Randy Eckenrode

There’s a tmp_cleaner launch daemon that runs /usr/libexec/tmp_cleaner. It doesn’t appear to be configurable. Oh, it’s actually in the diskdev_cmds source release. It’s just a shell script and definitely not configurable.

https://github.com/apple-oss-distributions/diskdev_cmds/blob/main/tmp_cleaner/tmp_cleaner Once diskdev_cmds is updated, it might be possible to use that script to do the cleanup. It would require > setting up a launch agent for it and patching the script with the new store’s tmp path.

That said, the tmp_cleaner script itself looks fairly simple and leans on both sed/find, so it'd probably make more sense to just build this into Nix than to depend on patching what's in diskdev_cmds.

[lilyball]

When we've seen this come up in the macOS room, I think it was reported (maybe by @lilyball?) that there isn't a way to convert the volume.

It's been a while since I looked into this, but yeah my recollection is there's no way to in-place convert an APFS (Case-insensitive) volume to an APFS (Case-sensitive) volume, the only conversion is to go from HFS to APFS.

That said, I'd love to see an upgrade process where it creates a new volume in the same APFS container, and then copies the entire store over (or at least the items reachable from roots, no sense copying garbage), verifies everything looks good, swaps any other setup over to pointing at the new volume, kills nix-daemon, unmounts the old store, remounts the new store at /nix, and relaunches nix-daemon. This is very complicated though, and of course "uninstall and reinstall" is vastly simpler, but for users heavily dependent on Nix it may not be feasible to uninstall. I certainly had a lot of trouble doing the manual migration myself given my heavy usage of nix-darwin.

Also worth noting, any copying of the old store to the new needs to be done by Nix itself rather than just copying the FS, as it has to handle the case hacked files. Although even that is problematic because Nix has ways of getting files into the store that bypasses the case hack mechanism (e.g. last I checked flake inputs do this, which is why having the linux kernel sources as a flake input broke things for me and I had to manually delete that whole store path and redownload it).

Also on the topic of the tmpdir, it absolutely needs to be moved onto the volume. Not just for nix-daemon, but also for every user invocation of nix and nix-*, because it's not just used for builds, it's also used as the initial download location for things (flake inputs for sure, and maybe builtin fetches though I haven't tested). My local setup involves not only setting services.nix-daemon.tempDir = "/nix/tmp" but also setting nix.package to a buildEnv that uses makeBinaryWrapper to wrap every file in ${nix}/bin/ to set TMPDIR to /nix/tmp (in retrospect, with that wrapper, setting services.nix-daemon.tempDir is a bit redundant).

[lf-]

lix equivalent issue: https://git.lix.systems/lix-project/lix/issues/332

since we now have build-dir, we could put the build directory in /nix/var/tmp. this would put it on the same fs as the store (maybe making certain things faster), deal with /tmp being tmpfs on many linux, and in general make things less busted. Randy reports case sensitive store is mostly not broken.

[emilazy]

Linking https://github.com/NixOS/nixpkgs/pull/347636#issuecomment-2406075655 with some discussion about what needs to be done for this. build-dir is insufficient.

[Enzime]

I was thinking about whether we would need the case hack for Windows, but it seems like they ran into similar issues of compatibility with WSL so they added an extended attribute to make a directory tree case sensitive (https://learn.microsoft.com/en-us/windows/wsl/case-sensitivity) so we can make the Nix Store on Windows case sensitive by default