NixOS / nix

Nix, the purely functional package manager
https://nixos.org/
GNU Lesser General Public License v2.1
12.14k stars 1.47k forks source link

Volume failed to mount: Nix fails to install on macOS if a partial install already created a `Nix Store` APFS volume #11426

Open 5HT2 opened 1 week ago

5HT2 commented 1 week ago

Platform

$ uname -a
Darwin apricot 23.6.0 Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:30 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6000 arm64 arm Darwin

Additional information

I had previously ran a Nix install, it was started implicitly by another program, once I realized it was doing so I cancelled it and decided to install Nix the way the proper documentation states (and so I could do it interactively and see what it was doing).

$ sh <(curl -L https://nixos.org/nix/install)

### Shown below ###

$ diskutil list
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.3 GB   disk0
   1:             Apple_APFS_ISC Container disk1         524.3 MB   disk0s1
   2:                 Apple_APFS Container disk3         494.4 GB   disk0s2
   3:        Apple_APFS_Recovery Container disk2         5.4 GB     disk0s3

/dev/disk3 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +494.4 GB   disk3
                                 Physical Store disk0s2
   1:                APFS Volume Macintosh HD            10.3 GB    disk3s1
   2:              APFS Snapshot com.apple.os.update-... 10.3 GB    disk3s1s1
   3:                APFS Volume Preboot                 6.1 GB     disk3s2
   4:                APFS Volume Recovery                936.4 MB   disk3s3
   5:                APFS Volume Data                    444.1 GB   disk3s5
   6:                APFS Volume VM                      6.4 GB     disk3s6
   7:                APFS Volume Nix Store               24.6 KB    disk3s7
   8:                APFS Volume Nix Store               24.6 KB    disk3s8

$ diskutil apfs unlockVolume disk3s7
Passphrase:
Unlocking any cryptographic user on APFS Volume disk3s7
Passphrase incorrect or user does not exist

$ diskutil info 'Nix Store'
   Device Identifier:         disk3s8
   Device Node:               /dev/disk3s8
   Whole:                     No
   Part of Whole:             disk3

   Volume Name:               Nix Store
   Mounted:                   No

   Partition Type:            41504653-0000-11AA-AA11-00306543ECAC
   File System Personality:   APFS
   Type (Bundle):             apfs
   Name (User Visible):       APFS
   Owners:                    Disabled

   OS Can Be Installed:       No
   Booter Disk:               disk3s2
   Recovery Disk:             disk3s3
   Media Type:                Generic
   Protocol:                  Apple Fabric
   SMART Status:              Verified
   Volume UUID:               25527FD8-7494-4069-8FCC-6B7A0A08A2BD
   Disk / Partition UUID:     25527FD8-7494-4069-8FCC-6B7A0A08A2BD

   Disk Size:                 494.4 GB (494384795648 Bytes) (exactly 965595304 512-Byte-Units)
   Device Block Size:         4096 Bytes

   Volume Used Space:         24.6 KB (24576 Bytes) (exactly 48 512-Byte-Units)
   Container Total Space:     494.4 GB (494384795648 Bytes) (exactly 965595304 512-Byte-Units)
   Container Free Space:      26.4 GB (26364313600 Bytes) (exactly 51492800 512-Byte-Units)

   Media OS Use Only:         No
   Media Read-Only:           No
   Volume Read-Only:          Not applicable (not mounted)

   Device Location:           Internal
   Removable Media:           Fixed

   Solid State:               Yes
   Hardware AES Support:      Yes

   This disk is an APFS Volume.  APFS Information:
   APFS Container:            disk3
   APFS Physical Store:       disk0s2
   Fusion Drive:              No
   FileVault:                 No
   Sealed:                    No
   Locked:                    No

$ sudo cat /etc/fstab
#
# Warning - this file should only be modified with vifs(8)
#
# Failure to do so is unsupported and may be destructive.
#
UUID=25527FD8-7494-4069-8FCC-6B7A0A08A2BD /nix apfs rw,noauto,nobrowse,suid,owners

Additional Notes

Proposed solution

It looks like encrypt_volume() in create-darwin-volume.sh is responsible for this.

If we take a look at setup_fstab() (which had the correct UUID) we can see that it just uses volume_uuid variable, when referring to it. We can also see that encrypt_volume() has access to the UUID, in addition to the volume label:

Relevant code snippet ```sh # Comments have been trimmed for brevity encrypt_volume() { local volume_uuid="$1" local volume_label="$2" local password task "Encrypt the Nix volume" >&2 _sudo "to mount your Nix volume for encrypting" \ /usr/sbin/diskutil mount "$volume_label" password="$(/usr/bin/xxd -l 32 -p -c 256 /dev/random)" _sudo "to add your Nix volume's password to Keychain" \ /usr/bin/security -i <

man diskutil tells us this:

manpage snippets ``` * For more * information on specifying device arguments, see the DEVICES section below. [...] * mount [readOnly] [nobrowse] [-mountOptions option [, option]] [-mountPoint path] device [...] * DEVICES A device parameter for any of the above commands (except where explicitly required otherwise) can usually be any of the following: • The disk identifier (see below). Any entry of the form of disk*, e.g. disk1s9. • The device node entry containing the disk identifier. Any entry of the form of /dev/[r]disk*, e.g. /dev/disk2. • The volume mount point. Any entry of the form of /Volumes/*, e.g. /Volumes/Untitled. In most cases, a "custom" mount point e.g. /your/custom/mountpoint/here is also accepted. • The URL form of any of the volume mount point forms described above. E.g. file:///Volumes/Untitled or file:///. • A UUID. Any entry of the form of e.g. 11111111-2222-3333-4444-555555555555. The UUID can be a "media" UUID which IOKit places in an IOMedia node as derived from e.g. a GPT map's partition UUID, or it can be an AppleRAID (or CoreStorage) set (LV) or member (PV) UUID. • A volume name, e.g. Untitled. This match is only attempted if the given device is not of the form [/dev/][r]disk*, nor [/Volumes/]*. The match attempt is against the intrinsic volume label, not against the terminal component, if mounted, of its mount point. ```

I would like to propose a few changes to the macOS installer script to avoid this issue:

  1. Use UUID when referring to the volume for mount / unmount commands
  2. Verify that no other volumes labelled Nix Store exist
    1. Attempt to unmount other volumes labelled Nix Store when detecting an old Nix install
    2. Ask the user what to do with them, provide options to automatically cleanup or cancel the installation

Output

Output ```sh $ sh <(curl -L https://nixos.org/nix/install) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 4267 100 4267 0 0 10251 0 --:--:-- --:--:-- --:--:-- 154k downloading Nix 2.24.5 binary tarball for aarch64-darwin from 'https://releases.nixos.org/nix/nix-2.24.5/nix-2.24.5-aarch64-darwin.tar.xz' to '/var/folders/_y/m3fxlmfn2yx5vn0crvqn28gr0000gn/T/nix-binary-tarball-unpack.mIFsy0knLb'... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 14.6M 100 14.6M 0 0 3206k 0 0:00:04 0:00:04 --:--:-- 3206k Switching to the Multi-user Installer Welcome to the Multi-User Nix Installation This installation tool will set up your computer with the Nix package manager. This will happen in a few stages: 1. Make sure your computer doesn't already have Nix. If it does, I will show you instructions on how to clean up your old install. 2. Show you what I am going to install and where. Then I will ask if you are ready to continue. 3. Create the system users (uids [301..332]) and groups (gid 30000) that the Nix daemon uses to run builds. To create system users in a different range, exit and run this tool again with NIX_FIRST_BUILD_UID set. 4. Perform the basic installation of the Nix files daemon. 5. Configure your shell to import special Nix Profile files, so you can use Nix. 6. Start the Nix daemon. Would you like to see a more detailed list of what I will do? [y/n] y I will: - make sure your computer doesn't already have Nix files (if it does, I will tell you how to clean them up.) - create local users (see the list above for the users I'll make) - create a local group (nixbld) - install Nix in /nix - create a configuration file in /etc/nix - set up the "default profile" by creating some Nix-related files in /var/root - back up /etc/bashrc to /etc/bashrc.backup-before-nix - update /etc/bashrc to include some Nix configuration - back up /etc/zshrc to /etc/zshrc.backup-before-nix - update /etc/zshrc to include some Nix configuration - create a Nix volume and a LaunchDaemon to mount it - create a LaunchDaemon (at /Library/LaunchDaemons/org.nixos.nix-daemon.plist) for nix-daemon Ready to continue? [y/n] y ---- let's talk about sudo ----------------------------------------------------- This script is going to call sudo a lot. Every time I do, it'll output exactly what it'll do, and why. Just like this: ---- sudo execution ------------------------------------------------------------ I am executing: $ sudo echo to demonstrate how our sudo prompts look This might look scary, but everything can be undone by running just a few commands. I used to ask you to confirm each time sudo ran, but it was too many times. Instead, I'll just ask you this one time: Can I use sudo? [y/n] y Yay! Thanks! Let's get going! ~~> Fixing any leftover Nix volume state Before I try to install, I'll check for any existing Nix volume config and ask for your permission to remove it (so that the installer can start fresh). I'll also ask for permission to fix any issues I spot. /dev/fd/63:1: parser error : Start tag expected, '<' not found ger> ^ unable to parse /dev/fd/63 During install, I add 'nix' to /etc/synthetic.conf, which instructs macOS to create an empty root directory for mounting the Nix volume. Can I remove /etc/synthetic.conf? [y/n] y ---- sudo execution ------------------------------------------------------------ I am executing: $ sudo rm /etc/synthetic.conf to remove /etc/synthetic.conf Password: During install, I add '/nix' to /etc/fstab so that macOS knows what mount options to use for the Nix volume. Can I remove /etc/fstab? [y/n] y ---- sudo execution ------------------------------------------------------------ I am executing: $ sudo rm /etc/fstab to remove /etc/fstab ~~> Checking for artifacts of previous installs Before I try to install, I'll check for signs Nix already is or has been installed on this system. ---- Nix config report --------------------------------------------------------- Temp Dir: /var/folders/_y/m3fxlmfn2yx5vn0crvqn28gr0000gn/T/tmp.4LnXaCxXUe Nix Root: /nix Build Users: 32 Build Group ID: 30000 Build Group Name: nixbld build users: Username: UID _nixbld1: 301 _nixbld2: 302 _nixbld3: 303 _nixbld4: 304 _nixbld5: 305 _nixbld6: 306 _nixbld7: 307 _nixbld8: 308 _nixbld9: 309 _nixbld10: 310 _nixbld11: 311 _nixbld12: 312 _nixbld13: 313 _nixbld14: 314 _nixbld15: 315 _nixbld16: 316 _nixbld17: 317 _nixbld18: 318 _nixbld19: 319 _nixbld20: 320 _nixbld21: 321 _nixbld22: 322 _nixbld23: 323 _nixbld24: 324 _nixbld25: 325 _nixbld26: 326 _nixbld27: 327 _nixbld28: 328 _nixbld29: 329 _nixbld30: 330 _nixbld31: 331 _nixbld32: 332 Ready to continue? [y/n] y ---- Preparing a Nix volume ---------------------------------------------------- Nix traditionally stores its data in the root directory /nix, but macOS now (starting in 10.15 Catalina) has a read-only root directory. To support Nix, I will create a volume and configure macOS to mount it at /nix. ~~> Configuring /etc/synthetic.conf to make a mount-point at /nix ---- sudo execution ------------------------------------------------------------ I am executing: $ sudo /usr/bin/ex -u NONE -n /etc/synthetic.conf to add Nix to /etc/synthetic.conf ~~> Creating a Nix volume ---- sudo execution ------------------------------------------------------------ I am executing: $ sudo /usr/sbin/diskutil apfs addVolume disk3 APFS Nix Store -nomount to create a new APFS volume 'Nix Store' on disk3 ---- sudo execution ------------------------------------------------------------ I am executing: $ sudo /usr/sbin/diskutil unmount force disk3s8 to ensure the Nix volume is not mounted disk3s8 was already unmounted ~~> Configuring /etc/fstab to specify volume mount options ---- sudo execution ------------------------------------------------------------ I am executing: $ sudo /usr/sbin/vifs to add nix to fstab ~~> Encrypt the Nix volume ---- sudo execution ------------------------------------------------------------ I am executing: $ sudo /usr/sbin/diskutil mount Nix Store to mount your Nix volume for encrypting Volume on disk3s7 failed to mount This is an encrypted and locked APFS Volume; use "diskutil apfs unlockVolume" ---- oh no! -------------------------------------------------------------------- Oh no, something went wrong. If you can take all the output and open an issue, we'd love to fix the problem so nobody else has this issue. :( We'd love to help if you need it. You can open an issue at https://github.com/NixOS/nix/issues/new?labels=installer&template=installer.md Or get in touch with the community: https://nixos.org/community ```

Priorities

Add :+1: to issues you find important.

abathur commented 1 week ago

I would like to propose a few changes to the macOS installer script to avoid this issue: ...

  1. Verify that no other volumes labelled Nix Store exist
  2. Attempt to unmount other volumes labelled Nix Store when detecting an old Nix install
  3. Ask the user what to do with them, provide options to automatically cleanup or cancel the installation

There already is a routine, mostly implemented by:

https://github.com/NixOS/nix/blob/668d63d8dd68d5999a9c9e79f265038ac8ce739c/scripts/create-darwin-volume.sh#L144-L197

and

https://github.com/NixOS/nix/blob/668d63d8dd68d5999a9c9e79f265038ac8ce739c/scripts/create-darwin-volume.sh#L555-L620

I suspect from this part of your log:

~~> Fixing any leftover Nix volume state
Before I try to install, I'll check for any existing Nix volume config
and ask for your permission to remove it (so that the installer can
start fresh). I'll also ask for permission to fix any issues I spot.
/dev/fd/63:1: parser error : Start tag expected, '<' not found
ger>
^
unable to parse /dev/fd/63

that it actually is running into trouble trying to run that routine, particularly in:

https://github.com/NixOS/nix/blob/668d63d8dd68d5999a9c9e79f265038ac8ce739c/scripts/create-darwin-volume.sh#L92-L111

Can you try copying that volumes_labeled() function into your shell (or an instance of bash, if you are running some other shell), and then confirm that volumes_labeled "Nix Store" yields the same error?

It's intended to print something like:

$ volumes_labeled "Nix Store"
disk1s6=B8F5A627-8288-45B5-959A-A9BA57594593

The error itself is making me wonder if the format of /usr/sbin/ioreg -ra -c "AppleAPFSVolume" has changed.

Edit: maybe also worth going ahead and collecting the output of these commands:

type -a xsltproc
xsltproc --version
5HT2 commented 1 week ago

Unfortunately, I had already manually deleted the volume by the time I posted the bug report, I will attempt to re-create making two volumes and try that?

abathur commented 1 week ago

The bits I asked you to try shouldn't require a 2nd volume (unless there was something odd about the first volume I guess).