search

NixOS / nix

Nix, the purely functional package manager
https://nixos.org/
GNU Lesser General Public License v2.1
12.92k stars 1.53k forks source link

/nix will not be writable on macOS Catalina #2925

Closed mroi closed 4 years ago

mroi commented 5 years ago

This is not a short term bug, but it will become an issue when macOS Catalina is released this fall. macOS is now split across two volumes (system and data) with a read-only system volume. This means that /nix will no longer be writable.

Some more information can be found in the related WWDC talk and some session notes people took from a Q&A.

Summary: the system volume, which is mounted at / will become non-writable. Some directories that need to be writable are connected via firmlinks (an Apple invention) to the data volume. /nix is not among these locations, so with the release of macOS Catalina, this location is no longer an option.

I see two possible solutions:

  1. I could try and file a bug to convince Apple to pre-install /nix as a firmlink to a writable location. I think this has limited success and Nix would then depend on Apple to not drop this link in a future release.
  2. We could move Nix on macOS to a different default location. Possible locations are those that Apple chooses to pre-install as links to writable locations. Two examples are /usr/local and /opt, so we could move to /usr/local/nix or /opt/nix. I would hope that these locations are common enough so that Apple would not drop them in the future.

I wanted to raise this issue early, before it becomes a problem for users. If this issue tracker is not the right place, please feel free to move this discussion elsewhere. I would also be available for testing any potential solution, since I have access to a macOS Catalina beta.

edolstra commented 5 years ago

Realistically, this may force us to drop support for macOS. Using a different store location for macOS would require a separate binary cache and a separate Hydra instance, and there would be no guarantee that the new location wouldn't break in the future.

I did notice in the talk that it's still possible to make the system volume writable, though not in a persistent way.

mroi commented 5 years ago

Speaking for myself: I would be quite sad if macOS support would go away. I will file a bug with Apple and see what happens. Cannot hurt.

Yes, you can jump through hoops and get a writable system volume, but only until the next reboot, so this is not a useful option.

jwiegley commented 5 years ago

@edolstra Before canceling support, we can ask the community if it's willing to host its own binary cache and Hydra instance, just as we've enlisted a host of volunteers to keep nixpkgs at near parity on Darwin. Having the answer be, "This doesn't easily fit our model so we won't support Mac" would be an unfortunate response.

mroi commented 5 years ago

I think no such decision should be made lightly, but it is definitely a drastic change on Apple’s part.

edolstra commented 5 years ago

Yes, you can jump through hoops and get a writable system volume, but only until the next reboot, so this is not a useful option.

Maybe you can use this to create a /nix symlink to another volume, which would hopefully persist across reboots?

mroi commented 5 years ago

That was my first try to get it temporarily working again. But nix-env complains then:

error: cannot open connection to remote store 'daemon': the path '/nix' is a symlink; this is not allowed for the Nix store and its parent directories

If that error can be safely ignored, I think this would be a workaround worth exploring. The way I understand things, anything you place in the root folder would persist until the next system update.

LnL7 commented 5 years ago

There's no real reason for the nix store to be on the system volume, with apfs it's easy to create a separate volume and mount it to /nix. But perhaps there's something I'm missing because I don't understand the need for these firmlinks, the equivalent is possible with mounts similar to how the store is mounted readonly on NixOS.

edolstra commented 5 years ago

@mroi You can set NIX_IGNORE_SYMLINK_STORE=1 to disable that check. (This should probably be turned into a nix.conf option.)

jwiegley commented 5 years ago

@edolstra Great idea, added #2926.

mroi commented 5 years ago

@edolstra Thanks for the tip, this is a useful workaround.

But future Nix-users on macOS still need to find a reliable way to inject this symlink into the read-only root directory. This probably involves some advanced steps like rebooting into recovery mode after every system upgrade.

@LnL7 The same would be true for /nix as a mount point, because we would need to create this directory on the read-only volume.

LnL7 commented 5 years ago

Sure but if Apple doesn't want to provide some sort of solution for one of these cases there will be no alternative to changing the default prefix. Which is arguably is not worth it for a secondary platform that's continuously moving further away from UNIX.

Is there an overview of the full volume layout somewhere? /usr/local and /Users are used as examples but I can't imagine that's everything.

kevingriffin commented 5 years ago

@mroi How have you been testing this? In my testing, I was able to install nix, but I assumed it was due to the comment in the presentation that "system volume is writeable in the developer preview". I'd also like to help test, but was able to get nix installed:


kevingriffin@KevinnoiMac /nix % uname -a
Darwin KevinnoiMac.local 19.0.0 Darwin Kernel Version 19.0.0: Fri May 24 17:36:10 PDT 2019; root:xnu-6041.0.0.111.5~1/RELEASE_X86_64 x86_64
kevingriffin@KevinnoiMac /nix % ls /nix
store   var
kevingriffin@KevinnoiMac /nix % nix-env -iA nixpkgs.hello
installing 'hello-2.10'
these paths will be fetched (0.02 MiB download, 0.07 MiB unpacked):
  /nix/store/c4w9z1kzzkdsgvgr6cy9ggl39s2yzn70-hello-2.10
copying path '/nix/store/c4w9z1kzzkdsgvgr6cy9ggl39s2yzn70-hello-2.10' from 'https://cache.nixos.org'...
building '/nix/store/iwcsfsh5hxyzpymnkp1r54nb3zzpd7mg-user-environment.drv'...
created 2 symlinks in user environment```
matthewbauer commented 5 years ago

It looks like you can still access the /nix directory after the update, it's just in /System/Volumes/Data/nix. Too bad there's no chroot equivalent on macOS.

matthewbauer commented 5 years ago

sudo mount -uw / && sudo ln -s /System/Volumes/Data/nix /nix works for me! This is on SIP even! Having the installer handle this might work.

kevingriffin commented 5 years ago

I wonder if that will continue to work in the later builds, when they plan to enforce the read-only aspect.

mroi commented 5 years ago

Apple says in the WWDC talk, that the first beta seeds mount the system volume read-write, but this is only temporary. A later seed will turn it read-only. You can create a file /.rootro to opt-in to the read-only mount now.

mroi commented 5 years ago

But there is good news. I talked to an Apple engineer on Twitter and they are working on a solution to create symlinks in the root directory even when the system volume is mounted read-only. I guess we’ll just wait and see how this turns out.

For now, symlinking /nix to /System/Volumes/Data/nix in combination with NIX_IGNORE_SYMLINK_STORE=1 is a viable workaround for everyone on the macOS Catalina testing train.

angerman commented 5 years ago

@grahamc remember our discussion about symlink /nix and the installer during ZuriHac? Turns out, if I had read the issue tracker more closely I had known about NIX_IGNORE_SYMLINK_STORE=1, and wouldn't have been as annoyed.

matthewbauer commented 5 years ago

This appears to be fixed in 10.15 Beta 3! I can create a /nix directory and use it as a store, while /System, /bin, /sbin, etc. are still read only. Users upgrading from previous beta versions can do:

sudo rm -f /nix
sudo mv /System/Volumes/Data/nix /nix

then start the daemon if you are in multi user:

sudo launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist
sudo launchctl start org.nixos.nix-daemon

New installs should work out of the box.

SIGNOISE commented 5 years ago

/ will ship as non-writeable -- what you're seeing is a known issue where it can become writeable due to another system component. If you reboot, you will not see this work.

matthewbauer commented 5 years ago

/ will ship as non-writeable -- what you're seeing is a known issue where it can become writeable due to another system component. If you reboot, you will not see this work.

Will sudo mount -uw / continue to work?

SIGNOISE commented 5 years ago

Will sudo mount -uw / continue to work?

If you disable SIP, yes.

matthewbauer commented 5 years ago

Will sudo mount -uw / continue to work?

If you disable SIP, yes.

Ugh, I don't think we want to recommend that. Are any exceptions for Nix possible? I saw that Fink's /sw is included in root now, if we could get something like this for Nix it would be extremely helpful.

To be clear, changing the root directory of Nix from /nix to something else is possible, but also very painful. We would lose some resource sharing abilities between macOS & Linux, as well as require setting up a special binary cache. The only alternative to this is recommending to our users insecure practices like disabling SIP.

SIGNOISE commented 5 years ago

We're working on something that will allow you to create either a symlink or empty directory at / even though it's read-only. So you could go one of two ways:

  1. Create /nix symlink to the data volume
  2. Put the repo on a separate apfs volume that space shares within the container, create a /nix empty directory, and then mount that volume on that directory
matthewbauer commented 5 years ago

We're working on something that will allow you to create either a symlink or empty directory at / even though it's read-only. So you could go one of two ways:

  1. Create /nix symlink to the data volume
  2. Put the repo on a separate apfs volume that space shares within the container, create a /nix empty directory, and then mount that volume on that directory

Sounds reasonable! The main concern with (1) is that realpath(3) will still report /System/Volumes/Data/nix when pwd=/nix (https://github.com/NixOS/nix/issues/2926). That can get /System/Volumes/Data/nix hardcoded for some build systems. This could be a problem when that directory exists on Catalina but not on other machines. We need something a little bit stronger than a symlink ;)

mroi commented 5 years ago

If we were to mount Nix from a separate volume, we could use the opportunity to use the case-sensitive version of APFS. This would remove the need to fix weirdo packages relying on files that just differ in case.

SIGNOISE commented 5 years ago

I never object to a change which spreads the case-sensitivity gospel.

angerman commented 5 years ago

If we were to mount Nix from a separate volume, we could use the opportunity to use the case-sensitive version of APFS. This would remove the need to fix weirdo packages relying on files that just differ in case.

I have been doing this with great success (although with hfs+ case sensitive) with an external ssd for a while now. The primary motivation being, that I can take my nix store from my desktop with me when I use my laptop while walking about.

kirelagin commented 5 years ago

I have upgraded to a new beta, and at the beginning of the installation procedure, I got this message:

“Macintosh HD” contains files on the root of the volume. If you continue installation, these files will be deleted.

Sure enough, after the installation my /nix symlink was gone, so I had to recreate it manually. Once remounting rw stops working, this means doing the “disable SIP”, “remount”, “restore the symlink”, “enable SIP” after each OS upgrade. Not cool :(.

kirelagin commented 5 years ago

(btw, it seems that / is rw with this beta, no matter how many times I reboot 🤔)

mroi commented 5 years ago

I just wanted to update this issue with my current experiences: I have been running with the option of symlinking /nix to /System/Volumes/Data/nix for a while and I do not recommend it.

Some builds apparently resolve symlinks, so impure paths end up in build results. clang for example wants to link the LTO library from /System/Volumes/Data/nix, resulting in impure linkage errors from the ld wrapper. My user environment even contained some symlinks to /System/Volumes/Data/nix.

I therefore have now switched to a separate volume mounted at /nix and this works a lot better. I also placed the build directory there to benefit from case sensitivity. I have stressed the system a bit (by building a Linux cross-compiler on macOS) and so far everything works out.

tl;dr: I suggest we use a separate volume on macOS (or at least offer an option in the installer). Here is how I set things up:

mkdir -m 555 /nix ; chgrp wheel /nix
diskutil apfs addVolume disk1 APFSX Nix -mountpoint /nix
diskutil enableOwnership /nix
chmod 755 /nix ; chown nix:nix /nix ; chflags hidden /nix
echo 'LABEL=Nix /nix apfs rw' >> /etc/fstab

The initial creation of the mount point would require some special sauce once the root volume is read-only. I hope that @EHYPERCHICKEN will advise us when the proper solution is available.

SIGNOISE commented 5 years ago

Should be available right now. You can use synthetic.conf(5) to create a virtual empty directory at / instead of a symlink. That can be your mount point. To do this, you just need to add the following entry:

nix
mroi commented 5 years ago

Thanks, I did not know about this facility. Works just fine.

matthewbauer commented 5 years ago

This looks like a good route to go on. I think we can add this to the install scripts:

with a conditional based on either root being read only or macOS version being >= 10.15. This sounds doable if anyone wants to work on it. Ideally we would have a migration path for users who just upgraded to 10.15.

LnL7 commented 5 years ago

I played with synthetic.conf a few weeks ago and while I encountered some oddities with apfs.util both a symlink and mount works after a reboot.

ryanbooker commented 5 years ago

This is a variation of @mroi's workaround that worked for me:

sudo vim /etc/synthetic.conf # do what @EHYPERCHICKEN said
sudo diskutil apfs addVolume disk1 APFSX Nix -mountpoint /nix
sudo diskutil enableOwnership /nix
sudo chown -R <username> /nix
sudo echo 'LABEL=Nix /nix apfs rw' >> /etc/fstab
curl https://nixos.org/nix/install | sh

Disabling SIP may be necessary in there somewhere, not sure. I had it disabled. But you can reenable it afterward and things will still work.

SIGNOISE commented 5 years ago

Disabling SIP should not be required. This should all work with it on; if it does not please file a bug.

mroi commented 5 years ago

I can confirm. I always have SIP enabled.

One question though: Is there an endorsed way of activating changes to /etc/synthetic.conf without reboot? I guess not, but it would certainly simplify the installer.

fuzz commented 5 years ago

chroot(8) is available on macOS. /usr/local is the Unix-correct place to install software not included with the base OS per hier(7) and Unix design.

offlinehacker commented 5 years ago

I think that fixing that nix store can be placed on any location is a long term goal we should pursuit.

ryanbooker commented 5 years ago

Disabling SIP should not be required. This should all work with it on; if it does not please file a bug.

Great. I wasn't sure. Because I had already tried other things before discovering this method, and SIP just happened to be off at the time.

7c6f434c commented 5 years ago

@offlinehacker I think the Nix store can be put at an arbitrary location at the cost of full rebuild, and few packages will complain as long as the path doesn't have any special characters. Changing the default location is a huge coordination problem, though.

offlinehacker commented 5 years ago

@7c6f434c i assume it's due Hydra and binary cache?

kirelagin commented 5 years ago

/usr/local is the Unix-correct place to install software not included with the base OS per hier(7) and Unix design.

Not really. It is the correct place to install things compiled with make install locally. For “structured” things, e.g. provided by a third-party package manager (like Nix 😉), /opt is the right place.

kirelagin commented 5 years ago

diskutil apfs addVolume disk1 APFSX Nix -mountpoint /nix

It looks like this command creates an unencrypted volume. I am having a little bit of a trouble figuring out how to add “cryptographic users” and how to make the OS keep the keys and users in sync on the new volume the way it does on the default ones.

mroi commented 5 years ago

You can use diskutil apfs encrypt <volume> -user disk to encrypt the volume after creating it.

kirelagin commented 5 years ago

@mroi Are you sure that is how it works? The help text is a little unclear, but my understanding is that it says that the special “disk” user should be used if I want to have a specific password for this specific disk, rather than granting access to it to my Open Directory users. And that’s actually what I see when I run the command you suggested: it asks me to provide a password for a new Disk user.

7c6f434c commented 5 years ago

@7c6f434c i assume it's due Hydra and binary cache?

And each binary-cache-using deployment would need to migrate to the new store location. Preferably at the same time as the binary cache migrates.

mroi commented 5 years ago

@kirelagin Right, I misread what you want to do. I don’t know a way to add crypto users to a self-created APFS volume. I think you would need to add the new volume to the volume group of the boot volume. But I would be afraid of doing this, as Apple may assume that this group contains exactly two volumes (the system and the data volume).

kirelagin commented 5 years ago

I don’t feel that comfortable with an unencrypted /nix volume, but I still can’t figure out how to properly enable FileVault on it. I tried a couple of things, but they all failed so far:

sh-3.2# fdesetup enable -verbose -user kirelagin -device /nix
fdesetup: device path = disk1s6
Enter the password for user 'kirelagin':
fdesetup removePersonalRecoveryKey deleteCryptoUserFromVolume error = -69594 (The crypto user was not found on the APFS Volume)Error: A problem occurred while trying to enable FileVault. (-50)
sh-3.2# diskutil apfs encryptVolume Nix -user CB400CC8-1264-43BB-B3AF-0F9FA81F7334
You must specify the Disk user when starting encryption with no existing users

So I have started the encryption with -user disk, giving some random password for now. Then I was hoping fdesetup would start to work, but no luck:

sh-3.2# fdesetup enable -verbose -user kirelagin -device /nix
fdesetup: device path = /nix
Error: FileVault is already On.
sh-3.2# fdesetup add -verbose -usertoadd kirelagin -device /nix
fdesetup: device path = /nix
Error: This command requires the current device location be set to the boot volume.

I have absolutely no clue what this last error means. I guess I’ll now give adding it to the volume group a go and hope it won’t screw up my system 🤷‍♂.