Nix daemon blocked by sandbox on macOS Monterey (on external SSD)

daniels commented 2 years ago

Describe the bug

After installing nix the nix daemon won't run since it's being blocked by sandboxing from opening files in the nix store. This is happening on macOS Monterey installed on and booted from an external SSD.

Steps To Reproduce

(Install and boot MacOS on an external SSD.)
Follow the installation instructions for Nix on MacOS: sh <(curl -L https://nixos.org/nix/install)
Complete the installation steps. (Notice that the Nix store volume was created on the external SSD.)
Check if nix is running: ps -ef | grep nix. It's not. (Occasionally a wait4path is running if darwin-store daemon hasn't started correctly.)
Check nix-daemon log: tail /private/var/log/nix-daemon.log

dyld[1397]: Library not loaded: /nix/store/95n0y1033piss1slf99is61a3zc0yv0p-libsodium-1.0.18/lib/libsodium.23.dylib
  Referenced from: /nix/store/hyfj5imsd0c4amlcjpf8l6w4q2draaj3-nix-2.7.0/bin/nix-daemon
  Reason: tried: '/nix/store/95n0y1033piss1slf99is61a3zc0yv0p-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/usr/local/lib/libsodium.23.dylib' (no such file), '/usr/lib/libsodium.23.dylib' (no such file)

(Repeated over and over again.)

So libsodium can't be loaded due to sandbox rules.

Expected behavior

nix-deamon and darwin-store running

nix-env --version output

-

Additional context

Note that I'm running MacOS on a FileVault encrypted external SSD (with the internal disk locked). I don't know for sure that this is related to the problem, but as it's probably not the most usual setup it might be the reason why I encounter this issue while others can install Nix on MacOS without problem.

abathur commented 2 years ago

I haven't run it on an external disk, but I would imagine someone has. I don't know what this would be off the top of my head, but I guess we can start picking at it. A few Qs:

What do these diskutil commands report?
- diskutil list
- diskutil apfs list
- diskutil info /
- diskutil info /nix
What happens if you invoke /nix/store/hyfj5imsd0c4amlcjpf8l6w4q2draaj3-nix-2.7.0/bin/nix-daemon directly from a terminal?
What does with the internal disk locked mean? The disk is encrypted and the credential isn't in keychain?

daniels commented 2 years ago

Hi, thanks for helping out.

Running sudo /nix/store/hyfj5imsd0c4amlcjpf8l6w4q2draaj3-nix-2.7.0/bin/nix-daemon worked. The deamon is running and I can use e.g. nix-shell -p hello. Probably should have checked this ... But I still don't know why it's different when it's started as a launch deamon.
It means exactly that. Encrypted with FileVault and the credentials not in the keychain.

.

Here is the output of each of the commands:

MacBook-Pro% diskutil list
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.3 GB   disk0
   1:                        EFI ⁨EFI⁩                     314.6 MB   disk0s1
   2:                 Apple_APFS ⁨Container disk1⁩         500.0 GB   disk0s2

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +500.0 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume ⁨Macintosh HD – data⁩     342.6 GB   disk1s1
   2:                APFS Volume ⁨Preboot⁩                 342.0 MB   disk1s2
   3:                APFS Volume ⁨Recovery⁩                626.5 MB   disk1s3
   4:                APFS Volume ⁨VM⁩                      6.4 GB     disk1s4
   5:                APFS Volume ⁨Macintosh HD⁩            15.3 GB    disk1s5
   6:                APFS Volume ⁨Nix Store⁩               29.7 GB    disk1s7

/dev/disk2 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.1 GB   disk2
   1:                        EFI ⁨EFI⁩                     209.7 MB   disk2s1
   2:                 Apple_APFS ⁨Container disk3⁩         499.9 GB   disk2s2

/dev/disk3 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +499.9 GB   disk3
                                 Physical Store disk2s2
   1:                APFS Volume ⁨Komprimatorn SSD – data⁩ 368.7 GB   disk3s1
   2:                APFS Volume ⁨Preboot⁩                 271.2 MB   disk3s2
   3:                APFS Volume ⁨Recovery⁩                1.1 GB     disk3s3
   4:                APFS Volume ⁨VM⁩                      8.6 GB     disk3s4
   5:                APFS Volume ⁨Komprimatorn SSD⁩        15.7 GB    disk3s5
   6:              APFS Snapshot ⁨com.apple.os.update-...⁩ 15.7 GB    disk3s5s1
   7:                APFS Volume ⁨Nix Store⁩               494.5 MB   disk3s7

MacBook-Pro% diskutil apfs list
APFS Containers (2 found)
|
+-- Container disk1 30815887-19CF-4A17-8E8E-169D1741EFB1
|   ====================================================
|   APFS Container Reference:     disk1
|   Size (Capacity Ceiling):      499963170816 B (500.0 GB)
|   Capacity In Use By Volumes:   395193638912 B (395.2 GB) (79.0% used)
|   Capacity Not Allocated:       104769531904 B (104.8 GB) (21.0% free)
|   |
|   +-< Physical Store disk0s2 3AFBE9E5-6BEE-4626-9865-11BB102F293E
|   |   -----------------------------------------------------------
|   |   APFS Physical Store Disk:   disk0s2
|   |   Size:                       499963170816 B (500.0 GB)
|   |
|   +-> Volume disk1s1 591DEFF8-8B2F-35D1-827A-2ED22C268720
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk1s1 (Data)
|   |   Name:                      Macintosh HD – data (Case-insensitive)
|   |   Mount Point:               Not Mounted
|   |   Capacity Consumed:         342629449728 B (342.6 GB)
|   |   Sealed:                    No
|   |   FileVault:                 Yes (Locked)
|   |
|   +-> Volume disk1s2 20AE12A9-A39E-476C-9361-DC34DC53021D
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk1s2 (Preboot)
|   |   Name:                      Preboot (Case-insensitive)
|   |   Mount Point:               Not Mounted
|   |   Capacity Consumed:         341970944 B (342.0 MB)
|   |   Sealed:                    No
|   |   FileVault:                 No
|   |
|   +-> Volume disk1s3 1938929C-1561-4F29-9BE7-EE93237D65CD
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk1s3 (Recovery)
|   |   Name:                      Recovery (Case-insensitive)
|   |   Mount Point:               Not Mounted
|   |   Capacity Consumed:         626454528 B (626.5 MB)
|   |   Sealed:                    No
|   |   FileVault:                 No
|   |
|   +-> Volume disk1s4 6FCCF702-7C1C-46A8-AACD-4449F226C3C5
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk1s4 (VM)
|   |   Name:                      VM (Case-insensitive)
|   |   Mount Point:               Not Mounted
|   |   Capacity Consumed:         6443610112 B (6.4 GB)
|   |   Sealed:                    No
|   |   FileVault:                 No
|   |
|   +-> Volume disk1s5 33E7EE67-CED5-4A70-9AAC-FF52BA1F13AB
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk1s5 (System)
|   |   Name:                      Macintosh HD (Case-insensitive)
|   |   Mount Point:               /Volumes/Macintosh HD
|   |   Capacity Consumed:         15321980928 B (15.3 GB)
|   |   Sealed:                    Yes
|   |   FileVault:                 Yes (Locked)
|   |   Encrypted:                 No
|   |
|   +-> Volume disk1s7 D0A98DB1-1115-4CEA-BE4D-86CED0EFBAAA
|       ---------------------------------------------------
|       APFS Volume Disk (Role):   disk1s7 (No specific role)
|       Name:                      Nix Store (Case-insensitive)
|       Mount Point:               Not Mounted
|       Capacity Consumed:         29667000320 B (29.7 GB)
|       Sealed:                    No
|       FileVault:                 Yes (Locked)
|
+-- Container disk3 5B1B8E0D-EB85-4019-A4D7-89AC7DDFA889
    ====================================================
    APFS Container Reference:     disk3
    Size (Capacity Ceiling):      499898105856 B (499.9 GB)
    Capacity In Use By Volumes:   395098595328 B (395.1 GB) (79.0% used)
    Capacity Not Allocated:       104799510528 B (104.8 GB) (21.0% free)
    |
    +-< Physical Store disk2s2 9F148CE8-A8D5-4747-9642-39FDFAE5A673
    |   -----------------------------------------------------------
    |   APFS Physical Store Disk:   disk2s2
    |   Size:                       499898105856 B (499.9 GB)
    |
    +-> Volume disk3s1 C138F899-4812-31CE-AADB-AD396FCBBA79
    |   ---------------------------------------------------
    |   APFS Volume Disk (Role):   disk3s1 (Data)
    |   Name:                      Komprimatorn SSD – data (Case-insensitive)
    |   Mount Point:               /System/Volumes/Data
    |   Capacity Consumed:         368734658560 B (368.7 GB)
    |   Sealed:                    No
    |   FileVault:                 Yes (Unlocked)
    |
    +-> Volume disk3s2 59691721-815D-42C8-9D1E-682FE212115C
    |   ---------------------------------------------------
    |   APFS Volume Disk (Role):   disk3s2 (Preboot)
    |   Name:                      Preboot (Case-insensitive)
    |   Mount Point:               /System/Volumes/Preboot
    |   Capacity Consumed:         271163392 B (271.2 MB)
    |   Sealed:                    No
    |   FileVault:                 No
    |
    +-> Volume disk3s3 AAF19554-2813-43CD-9A81-8F9464F7F4A4
    |   ---------------------------------------------------
    |   APFS Volume Disk (Role):   disk3s3 (Recovery)
    |   Name:                      Recovery (Case-insensitive)
    |   Mount Point:               Not Mounted
    |   Capacity Consumed:         1095483392 B (1.1 GB)
    |   Sealed:                    No
    |   FileVault:                 No
    |
    +-> Volume disk3s4 604F1841-C79B-40D7-94CB-1DFFCF7FAEC5
    |   ---------------------------------------------------
    |   APFS Volume Disk (Role):   disk3s4 (VM)
    |   Name:                      VM (Case-insensitive)
    |   Mount Point:               /System/Volumes/VM
    |   Capacity Consumed:         8591003648 B (8.6 GB)
    |   Sealed:                    No
    |   FileVault:                 No
    |
    +-> Volume disk3s5 FF642800-AB81-43D1-A9E2-2F55C97A2656
    |   ---------------------------------------------------
    |   APFS Volume Disk (Role):   disk3s5 (System)
    |   Name:                      Komprimatorn SSD (Case-insensitive)
    |   Mount Point:               Not Mounted
    |   Capacity Consumed:         15749931008 B (15.7 GB)
    |   Sealed:                    Yes
    |   FileVault:                 Yes (Unlocked)
    |   Encrypted:                 No
    |   |
    |   Snapshot:                  7A5E5ADC-C3EA-4C5E-9AE1-5AD8499F08CD
    |   Snapshot Disk:             disk3s5s1
    |   Snapshot Mount Point:      /
    |   Snapshot Sealed:           Yes
    |
    +-> Volume disk3s7 FF6E9075-BB92-4631-81DA-3481B13C2470
        ---------------------------------------------------
        APFS Volume Disk (Role):   disk3s7 (No specific role)
        Name:                      Nix Store (Case-insensitive)
        Mount Point:               /nix
        Capacity Consumed:         494452736 B (494.5 MB)
        Sealed:                    No
        FileVault:                 Yes (Unlocked)

MacBook-Pro% diskutil info /
   Device Identifier:         disk3s5s1
   Device Node:               /dev/disk3s5s1
   Whole:                     No
   Part of Whole:             disk3

   Volume Name:               Komprimatorn SSD
   Mounted:                   Yes
   Mount Point:               /

   Partition Type:            41504653-0000-11AA-AA11-00306543ECAC
   File System Personality:   APFS
   Type (Bundle):             apfs
   Name (User Visible):       APFS
   Owners:                    Enabled

   OS Can Be Installed:       No
   Booter Disk:               disk3s2
   Recovery Disk:             disk3s3
   Media Type:                Generic
   Protocol:                  USB
   SMART Status:              Not Supported
   Volume UUID:               7A5E5ADC-C3EA-4C5E-9AE1-5AD8499F08CD
   Disk / Partition UUID:     7A5E5ADC-C3EA-4C5E-9AE1-5AD8499F08CD

   Disk Size:                 499.9 GB (499898105856 Bytes) (exactly 976363488 512-Byte-Units)
   Device Block Size:         4096 Bytes

   Container Total Space:     499.9 GB (499898105856 Bytes) (exactly 976363488 512-Byte-Units)
   Container Free Space:      104.8 GB (104795762688 Bytes) (exactly 204679224 512-Byte-Units)
   Allocation Block Size:     4096 Bytes

   Media OS Use Only:         No
   Media Read-Only:           Yes
   Volume Read-Only:          Yes (read-only mount flag set)

   Device Location:           External
   Removable Media:           Fixed

   Solid State:               Yes

   This disk is an APFS Volume Snapshot.  APFS Information:
   APFS Snapshot Name:        com.apple.os.update-9D831579E656FF2E36BAC3334521407031CBA216B7079C5998A817A5EC7C76B7
   APFS Snapshot UUID:        7A5E5ADC-C3EA-4C5E-9AE1-5AD8499F08CD
   APFS Container:            disk3
   APFS Physical Store:       disk2s2
   Fusion Drive:              No
   APFS Volume Group:         C138F899-4812-31CE-AADB-AD396FCBBA79
   EFI Driver In macOS:       1933080003000000
   Encrypted:                 No
   FileVault:                 Yes
   Sealed:                    Yes
   Locked:                    No

   APFS Snapshots are defined upon this APFS Volume.  Snapshot list:
   Snapshot UUID:             7A5E5ADC-C3EA-4C5E-9AE1-5AD8499F08CD
   Name:                      com.apple.os.update-9D831579E656FF2E36BAC3334521407031CBA216B7079C5998A817A5EC7C76B7
   XID:                       50989

MacBook-Pro% diskutil info /nix
   Device Identifier:         disk3s7
   Device Node:               /dev/disk3s7
   Whole:                     No
   Part of Whole:             disk3

   Volume Name:               Nix Store
   Mounted:                   Yes
   Mount Point:               /nix

   Partition Type:            41504653-0000-11AA-AA11-00306543ECAC
   File System Personality:   APFS
   Type (Bundle):             apfs
   Name (User Visible):       APFS
   Owners:                    Enabled

   OS Can Be Installed:       Yes
   Booter Disk:               disk3s2
   Recovery Disk:             disk3s3
   Media Type:                Generic
   Protocol:                  USB
   SMART Status:              Not Supported
   Volume UUID:               FF6E9075-BB92-4631-81DA-3481B13C2470
   Disk / Partition UUID:     FF6E9075-BB92-4631-81DA-3481B13C2470

   Disk Size:                 499.9 GB (499898105856 Bytes) (exactly 976363488 512-Byte-Units)
   Device Block Size:         4096 Bytes

   Container Total Space:     499.9 GB (499898105856 Bytes) (exactly 976363488 512-Byte-Units)
   Container Free Space:      104.8 GB (104797212672 Bytes) (exactly 204682056 512-Byte-Units)
   Allocation Block Size:     4096 Bytes

   Media OS Use Only:         No
   Media Read-Only:           No
   Volume Read-Only:          No

   Device Location:           External
   Removable Media:           Fixed

   Solid State:               Yes

   This disk is an APFS Volume.  APFS Information:
   APFS Container:            disk3
   APFS Physical Store:       disk2s2
   Fusion Drive:              No
   FileVault:                 Yes
   Sealed:                    No
   Locked:                    No

gurk commented 2 years ago

I'm running into the same/similar issue.

Been through uninstall procedure (https://github.com/NixOS/nix/pull/6144) about 5 times.

Booting from external SSD, but not using FileVault. macOS Catalina.


  Referenced from: /nix/var/nix/profiles/default/bin/nix-daemon
  Reason: no suitable image found.  Did find:
    file system sandbox blocked open() of '/nix/store/jd1l64vr0b5y1qvdsrminn53gkvamm32-editline-1.17.1/lib/libeditline.1.dylib'
    /nix/store/jd1l64vr0b5y1qvdsrminn53gkvamm32-editline-1.17.1/lib/libeditline.1.dylib: stat() failed with errno=1
    file system sandbox blocked open() of '/nix/store/jd1l64vr0b5y1qvdsrminn53gkvamm32-editline-1.17.1/lib/libeditline.1.dylib'```

abathur commented 2 years ago

Still just kinda guessing/collecting.

I don't recall any ~sandbox reports like this before, but the external disk bit does remind me a little of some issues related to Nix installs on VMs and EC2. We don't fully understand it, but macOS does treat internal and external drives a little differently, so I guess that's the thread to pull on for now.

diskutil info -plist /nix
sudo vsdbutil -c /
sysadminctl -secureTokenStatus $USER

It might also be worth opening Console.app to see if there are any obviously-correlated messages that give a little more insight.

daniels commented 2 years ago

The darwin-store launch daemon seems to fail each boot. The following is after I run it manually with sudo launchctl kickstart -k system/org.nixos.darwin-store.

1.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>APFSContainerFree</key>
    <integer>134000287744</integer>
    <key>APFSContainerReference</key>
    <string>disk3</string>
    <key>APFSContainerSize</key>
    <integer>499898105856</integer>
    <key>APFSPhysicalStores</key>
    <array>
        <dict>
            <key>APFSPhysicalStore</key>
            <string>disk2s2</string>
        </dict>
    </array>
    <key>APFSSnapshot</key>
    <false/>
    <key>Bootable</key>
    <true/>
    <key>BooterDeviceIdentifier</key>
    <string>disk3s2</string>
    <key>BusProtocol</key>
    <string>USB</string>
    <key>CanBeMadeBootable</key>
    <false/>
    <key>CanBeMadeBootableRequiresDestroy</key>
    <false/>
    <key>Content</key>
    <string>41504653-0000-11AA-AA11-00306543ECAC</string>
    <key>DeviceBlockSize</key>
    <integer>4096</integer>
    <key>DeviceIdentifier</key>
    <string>disk3s7</string>
    <key>DeviceNode</key>
    <string>/dev/disk3s7</string>
    <key>DeviceTreePath</key>
    <string>IODeviceTree:/PCI0@0/PEG2@1,2/UPSB@0/DSB2@2/XHC3@0</string>
    <key>DiskUUID</key>
    <string>FF6E9075-BB92-4631-81DA-3481B13C2470</string>
    <key>Ejectable</key>
    <true/>
    <key>EjectableMediaAutomaticUnderSoftwareControl</key>
    <false/>
    <key>EjectableOnly</key>
    <false/>
    <key>Encryption</key>
    <true/>
    <key>EncryptionThisVolumeProper</key>
    <true/>
    <key>FileVault</key>
    <true/>
    <key>FilesystemName</key>
    <string>APFS</string>
    <key>FilesystemType</key>
    <string>apfs</string>
    <key>FilesystemUserVisibleName</key>
    <string>APFS</string>
    <key>FreeSpace</key>
    <integer>0</integer>
    <key>Fusion</key>
    <false/>
    <key>GlobalPermissionsEnabled</key>
    <true/>
    <key>IOKitSize</key>
    <integer>499898105856</integer>
    <key>IORegistryEntryName</key>
    <string>Nix Store</string>
    <key>Internal</key>
    <false/>
    <key>Locked</key>
    <false/>
    <key>MediaName</key>
    <string></string>
    <key>MediaType</key>
    <string>Generic</string>
    <key>MountPoint</key>
    <string>/nix</string>
    <key>OSInternalMedia</key>
    <false/>
    <key>ParentWholeDisk</key>
    <string>disk3</string>
    <key>PartitionMapPartition</key>
    <false/>
    <key>RAIDMaster</key>
    <false/>
    <key>RAIDSlice</key>
    <false/>
    <key>RecoveryDeviceIdentifier</key>
    <string>disk3s3</string>
    <key>Removable</key>
    <false/>
    <key>RemovableMedia</key>
    <false/>
    <key>RemovableMediaOrExternalDevice</key>
    <true/>
    <key>SMARTDeviceSpecificKeysMayVaryNotGuaranteed</key>
    <dict/>
    <key>SMARTStatus</key>
    <string>Not Supported</string>
    <key>Sealed</key>
    <string>No</string>
    <key>Size</key>
    <integer>499898105856</integer>
    <key>SolidState</key>
    <true/>
    <key>SupportsGlobalPermissionsDisable</key>
    <true/>
    <key>SystemImage</key>
    <false/>
    <key>TotalSize</key>
    <integer>499898105856</integer>
    <key>VolumeAllocationBlockSize</key>
    <integer>4096</integer>
    <key>VolumeName</key>
    <string>Nix Store</string>
    <key>VolumeSize</key>
    <integer>0</integer>
    <key>VolumeUUID</key>
    <string>FF6E9075-BB92-4631-81DA-3481B13C2470</string>
    <key>WholeDisk</key>
    <false/>
    <key>Writable</key>
    <true/>
    <key>WritableMedia</key>
    <true/>
    <key>WritableVolume</key>
    <true/>
</dict>
</plist>

2.

MacBook-Pro% sudo vsdbutil -c /
No entry found for '/'.

3.

MacBook-Pro% sysadminctl -secureTokenStatus $USER
2022-03-27 08:53:48.050 sysadminctl[5332:334253] Secure token is ENABLED for user NN

daniels commented 2 years ago

I have no experience at all with sandboxing on MacOS, except as an end user. However, something that showed up when I searched was that there seems to be two different privileges, SystemPolicyAllFiles and SystemPolicyRemovableVolumes, and that despite the naming (and description) the first one does not overlap with the second, e.g. both are needed. Is nix-daemon asking for both?

In my System preferences, under Security and Integrity -> Integrity -> Files and Folders, Terminal is listed as having "Removable Volumes", while nix-daemon is listed with "Full Disk Access". (I'm using Swedish so the terminology might be a bit different.) Maybe nix-daemon inherits the Terminal-privileges when I execute it manually?

abathur commented 2 years ago

Since we're exploring I'll try to think out loud for posterity (don't worry too much if something I say doesn't make sense; feel free to ask questions). I'll start with the command outputs (leaving your followup comment on the privileges aside for now):

I haven't been through this in detail yet, but the one item I was looking for here looks OK:
```
<key>GlobalPermissionsEnabled</key>
<true/>  
```
This one might be pointing to the key difference. My device (on the internal SSD) yields Permissions on '/' are enabled.
This looks OK as well.

The difference I note in 2 is out in that territory where we don't exactly understand what the system is doing well enough to be sure if it's a problem or not.

Our working thesis has been that we just need GlobalPermissionsEnabled to be true for the Nix volume, but perhaps the root volume also has to have the same.
We might be able to use vsdbutil to change this and see if it fixes the issue, but we're leaving that as a last resort because the manpage also indicates that vsdbutil is deprecated and tells us to use fstab instead.
instead we:
- use UUID=$uuid /nix apfs rw,noauto,nobrowse,suid,owners [source] (I think one or both of suid and owners attempt to address similar issues in ~VM contexts)
- more recently added something like /usr/sbin/diskutil enableOwnership /nix [source] after fstab alone didn't address the issue.

So my thought for the moment is:

Before we try anything, confirm that the value is false for the GlobalPermissionsEnabled key in the following command: diskutil info -plist /
If the value is indeed false, try running /usr/sbin/diskutil enableOwnership / and then see if the Nix volume mounts on reboot. (If this messes anything else up, you should be able to use /usr/sbin/diskutil disableOwnership / to reverse it.)

daniels commented 2 years ago

MacBook-Pro% diskutil info -plist / | grep -A1 GlobalPermissions
    <key>GlobalPermissionsEnabled</key>
    <true/>
--
    <key>SupportsGlobalPermissionsDisable</key>
    <true/>

So no use running enableOwnership, then?

Edit: Posted the wrong command execution. Output still the same.

abathur commented 2 years ago

Hmm. I was thinking not, but I just double-checked the manpage for diskutil, and it says enableOwnership will:

Enable ownership of a volume. The on-root-disk Volume Database at /var/db/volinfo.database is manipulated such that the User and Group ID settings of files, directories, and links (file system objects, or "FSOs") on the target volume are taken into account.

That's the same database vsdbutil would modify. We thought there was a good correlation between GlobalPermissionsEnabled == false and the need to run it, but that may be wrong.

So--yeah, go ahead and try sudo /usr/sbin/diskutil enableOwnership /

daniels commented 2 years ago

No difference after sudo /usr/sbin/diskutil enableOwnership / and reboot. The darwin-store service failed to mount the nix-volume until I launched it manually. And after that the same error in nix-daemon.log.

Since I had GlobalPermissionsEnabled before there should be no obvious need to reverse anything from the enableOwnership command?

abathur commented 2 years ago

Drat. Does sudo vsdbutil -c / now report Permissions on '/' are enabled.? And What does sudo vsdbutil -c /nix report?

daniels commented 2 years ago

MacBook-Pro% sudo vsdbutil -c /
Permissions on '/' are enabled.
MacBook-Pro% sudo vsdbutil -c /nix
No entry found for '/nix'.

abathur commented 2 years ago

Hmm. Can you try the enablePermissions command for /nix as well (and reboot)? I assumed the installer would have done that, but perhaps not.

daniels commented 2 years ago

I just did (including reboot), but still no difference, except for the output of sudo vsdbutil -c /nix which now reports Permissions on '/nix' are enabled.

daniels commented 2 years ago

I have no idea if it could have any impact, but I failed to mention when opening the issue that I had nix installed prior to upgrading to Monterey. So the re-installation failed asking me to remove nix entries in bashrc etc. I did all these things (repeatedly) until the installation went through without errors. While I removed the things I was asked to, I guess that e.g. build users created by the earlier installation might have been reused. (I did remove the new nix volume between each retry.)

abathur commented 2 years ago

Ah. Perhaps. A complete uninstall is tricky and might be confounding. Were you on Catalina? It was working with the same general setup?

It might be worth following the not-quite-released uninstall directions from the unstable manual https://nixos.org/manual/nix/unstable/installation/installing-binary.html#macos and giving a reinstall a try.

daniels commented 2 years ago

I was not on Catalina but went straight from Mojave to Monterey.

I now performed all the steps from the linked uninstall directions, followed by a reinstall of nix. The issue persists. The volume was mounted after installation but nix-daemon fails to access libsodium during so is not running. Will reboot now.

daniels commented 2 years ago

No luck. No mounted volume, and after manually triggering darwin-store, nix-daemon still fails.

abathur commented 2 years ago

Ok. Makes sense that you leapfrogged Catalina.

Back to your permission note earlier, nix doesn't currently use any of the ~Security permissions (and I don't think it can, automatically--IIUC the best anyone can do is trigger an action that would cause their app to show up in the list and then open the panel for the user to toggle it).

I would guess that you have added both of these permissions over the years while troubleshooting nix or other software, and the uninstall won't flush them. I think I recall setting full disk for the daemon being a workaround that the same set of EC2/VM users were using to get nix working before the more recent enableOwnership change (which at least appears to have caused the trickle of those reports to dry up).

Given that history, it does make sense to me that adding some may be forcing the issue. If you don't already have it installed with the same permission, it might be interesting to install iterm2 and see if manually invoking the daemon from there triggers the sandbox error, or if it just works?

Fair warning that I am otherwise running low on actionable hunches. I'll try to do a little more reading myself, but my responses may slow down a little if we hit another dead-end or two.

daniels commented 2 years ago

Thank you so much for all your effort so far! I wouldn't have expected anywhere near this engagement for an edge case like mine, but your response has been better than any paid customer service I have ever used ...

I would also like to thank you and your fellow committers for your work on the installer. It has become really great, with clear information on what's going on and helpful instructions when it detects something's off. It so good I almost don't mind having to reinstall nix for the fiftyeleventh time ...

I will test with iTerm2, but probably not before the weekend.

sheldonneuberger commented 2 years ago

Hitting the same issue. Manually running sudo -i /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix-daemon does seem to fix it. Anyone know the root cause here?

I'm seeing this on osx 12.4, running on an amazon mac2.metal instance (perhaps that's a way that people can repro this).

abathur commented 2 years ago

Hitting the same issue. Manually running sudo -i /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix-daemon does seem to fix it.

As in, running it once permanently fixes it? Or just that it works that time?

sheldonneuberger commented 2 years ago

It just works that time. So I leave that terminal window open to run the daemon, then in other terminal windows my nix-shell commands will work (I can see the daemon terminal outputting 'accepted connection').

abathur commented 2 years ago

What do stat /nix and stat /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix-daemon say?

sheldonneuberger commented 2 years ago

$ stat /nix
16777241 2 drwxr-xr-x 5 root nixbld 0 160 "Jul 26 23:32:17 2022" "Jul 26 23:15:51 2022" "Jul 26 23:15:51 2022" "Jul 26 23:15:33 2022" 4096 0 0 /nix

$ stat /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix-daemon
16777241 616 lr-xr-xr-x 1 ec2-user staff 0 3 "Jul 26 23:15:28 2022" "Jan  1 00:00:01 1970" "Jul 26 23:15:52 2022" "Jan  1 00:00:01 1970" 4096 0 0 /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix-daemon

$ stat /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix      
16777241 615 -r-xr-xr-x 1 ec2-user staff 0 3061456 "Jul 27 05:33:43 2022" "Jan  1 00:00:01 1970" "Jul 26 23:15:52 2022" "Jan  1 00:00:01 1970" 4096 5984 0 /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix

abathur commented 2 years ago

Two thoughts:

starting to wonder if this is a different (or maybe more modern?) manifestation of the problem reposted in #4640.
I doubt nixbld should be the group for /nix. I think maybe root is also a sign of trouble, per #4640, though I've overwithdrawn my brain bank for the day. I have an older install that I can't readily use as a reference for what yours should look like, but I'll ask on Matrix and see if someone with a working modern install can chime in...

sheldonneuberger commented 2 years ago

I have a physical mac where nix works fine (the above output is from a mac VM on ec2). Stat'ing the same files, I see on my working physical mac that root.nixbld owns /nix and everything in it.

So I think the problem on the mac VM is that the logged-in user is somehow gaining ownership of some files in the /nix/store (or perhaps the inverse like you said: the logged-in user should own everything, not root). Although it's unclear how this relates to the root cause problem of the fs sandbox blocking open() in /nix. Tomorrow, I'll try to manually chown /nix and see if it fixes the issue.

abathur commented 2 years ago

Someone on matrix chimed in and said they do have root/nixbld for both /nix and a copy of nix-daemon.

The source indicates that root/nixbld are correct.

(Someone on an older macos said they have root/wheel. Not sure what the cause of that one is...)

It's possible the installer is holding something wrong. I know it uses sudo a lot and I gather macos sudo is a little weird.

If basically everything else in your store is owned by your user (and the same isn't true on your physical Mac), I'd wonder first if aws is using a modified sudo that breaks our assumptions?

YorikSar commented 1 year ago

I've ran into the same issue on macOS Monterey 12.5: I moved /nix to external disk and re-juggled mount points, so that external disk was mounted as /nix. After that nix-daemon refused to start with file system sandbox blocked open() for libsodium:

 % sudo launchctl debug system/org.nixos.nix-daemon --stderr
Service configured for next launch.
dyld[59104]: Library not loaded: '/nix/store/3g5jq3wxcna446klkmrvvkbjzcy6ppcx-libsodium-1.0.18/lib/libsodium.23.dylib'
  Referenced from: '/nix/store/qzran14w2x48nk5dp0pji5rw0xj94r7s-nix-2.8.1/bin/nix'
  Reason: tried: '/nix/store/3g5jq3wxcna446klkmrvvkbjzcy6ppcx-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/usr/local/lib/libsodium.23.dylib' (no such file), '/usr/lib/libsodium.23.dylib' (no such file)

When I granted "Full Disk Access" to sh, this error disappeared and the daemon started just fine. Doing the same for nix binary didn't yield any changes, so I assume that the sandbox is built based on the binary that starts the daemon, not on the one that it exec()'ed into.

I wasn't happy with giving any user access to an executable with full disk access, so I:

made a copy of /bin/sh: sudo mkdir -p /usr/local/bin && sudo cp /bin/sh /usr/local/bin/shWithFullDiskAccess
allowed only root to run it: sudo chmod 400 /usr/local/bin/shWithFullDiskAccess
added it to "Full Disk Access" in "Privacy & Policy" settings tab
replaced /bin/sh with /usr/local/bin/shWithFullDiskAccess in /Library/LaunchDaemons/org.nixos.nix-daemon.plist

After all this, nix-daemon is running fine. It would be nice to have a small "starter" binary for nix-daemon that we could place on root partition instead though.

abathur commented 1 year ago

@YorikSar Am I understanding right that this is on a local system that's easy to poke at? (I don't have fresh ideas atm; just asking for the record since people able/willing to poke at this kind of issue on-demand have been scarce.)

YorikSar commented 1 year ago

@abathur Nope, it's a server-ish Mac that I spend whole day migrating to external SSD to have more space in Nix store :) So all poking requires a maintenance window and workflow disruption for others. I could find some old MacBook to play with though.

abathur commented 1 year ago

Drat :)

sheldonneuberger commented 1 year ago

It repro'd for me every time on an aws ec2 mac.

On Fri, Dec 23, 2022, 4:00 PM Travis A. Everett @.***> wrote:

Drat :)

— Reply to this email directly, view it on GitHub https://github.com/NixOS/nix/issues/6291#issuecomment-1364414786, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXDM2CM7XE4LOKRFBL345DWOY4JRANCNFSM5RFHZHFQ . You are receiving this because you commented.Message ID: @.***>

samrose commented 1 year ago

I confirmed that on x86_64-darwin macos ventura nix daemon runs without issue even with external SSD drive (this happens to be aws mac1.metal). However on arm/aarch64 darwin mac2.metal instance running ventura, we see the same issue reported here

nixos-discourse commented 1 year ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-04-14-nix-team-meeting-minutes-48/27358/1

abathur commented 1 year ago

@samrose @YorikSar @sheldonneuberger @bryanhonof Previously I mentioned opening Console.app to look for anything that correlates, and this morning I read a blog post that mentions sandbox denials being logged--so it's possible that some of you maybe be able to see a message there (or via the log CLI if you don't have GUI access) with a little more context?

(You should just be able to search "sandbox" to see all such messages, but narrowing the predicate or closing all other apps may make it easier to find the right ones. If you've added an exemption in Security & Privacy for these, you'll need to disable it and maybe reboot?)

I'm not quite sure it'll apply quite the same in our case since the blog post is specifically focused on apps, but https://ubrigens.com/posts/sandbox_coverage.html and the associated code in https://github.com/0xbf00/macos-sandbox-coverage may be a thread worth pulling on to see if we can at least identify the specific sandbox profile/rule that's snagging you.

If we know that, maybe we'll be able to walk this back to documentation or code that'll enable us to understand what's going on and whether this is intrinsic or something Nix or the installer could be working around.

YorikSar commented 1 year ago

I've created a macOS VM in UTM with macOS guest. Thankfully, UTM makes it extremely easy. I've created a file with truncate -s 10G VM-stick.img and added it as a removable disk to the VM. Then in VM passed it to the Nix installer via NIX_VOLUME_USE_SPECIAL=disk5s1 (I realise now I could've used NIX_VOLUME_USE_DISK=disk4 now). After installer ran, I see the same error in /var/log/nix-daemon.log as before. Ran Console.app with sandbox filter, got these lines pretty much repeating:

error   00:44:18.229674+0200    kernel  Sandbox: com.apple.WebKit.WebContent(643) deny(1) mach-lookup com.apple.diagnosticd
error   00:44:18.873198+0200    kernel  Sandbox: bluetoothd(128) deny(1) mach-lookup com.apple.private.corewifi-xpc
error   00:44:23.525302+0200    kernel  Sandbox: com.apple.WebKit.WebContent(660) deny(1) mach-lookup com.apple.diagnosticd
default 00:44:26.475768+0200    tccd    AUTHREQ_ATTRIBUTION: msgID=130.69, attribution={accessing={TCCDProcess: identifier=nix, pid=3179, auid=0, euid=0, binary_path=/nix/store/fzykws31cb5a88h3jkimzr944yaiqbyv-nix-2.15.0/bin/nix}, requesting={TCCDProcess: identifier=com.apple.sandboxd, pid=130, auid=0, euid=0, binary_path=/usr/libexec/sandboxd}, },
default 00:44:26.483561+0200    sandboxd    send_message_with_reply_sync(): user tccd unavailable, sending 0x11de0e180 to system tccd
default 00:44:26.484174+0200    tccd    AUTHREQ_ATTRIBUTION: msgID=130.70, attribution={accessing={TCCDProcess: identifier=nix, pid=3179, auid=0, euid=0, binary_path=/nix/store/fzykws31cb5a88h3jkimzr944yaiqbyv-nix-2.15.0/bin/nix}, requesting={TCCDProcess: identifier=com.apple.sandboxd, pid=130, auid=0, euid=0, binary_path=/usr/libexec/sandboxd}, },
error   00:44:26.484992+0200    kernel  System Policy: nix(3179) deny(1) file-read-data /nix/store/fzykws31cb5a88h3jkimzr944yaiqbyv-nix-2.15.0/bin/nix
error   00:44:26.485110+0200    kernel  System Policy: nix(3179) deny(1) file-read-data /nix/store/gbgryjc27mgd404q49ccjp6g5kf18c3v-libsodium-1.0.18/lib/libsodium.23.dylib
error   00:44:26.485136+0200    kernel  System Policy: nix(3179) deny(1) file-read-data /nix/store/gbgryjc27mgd404q49ccjp6g5kf18c3v-libsodium-1.0.18/lib
error   00:44:26.485163+0200    kernel  System Policy: nix(3179) deny(1) file-read-data /nix/store/gbgryjc27mgd404q49ccjp6g5kf18c3v-libsodium-1.0.18/lib/libsodium.23.dylib
error   00:44:26.485258+0200    kernel  2 duplicate reports for System Policy: nix(3179) deny(1) file-read-data /nix/store/gbgryjc27mgd404q49ccjp6g5kf18c3v-libsodium-1.0.18/lib/libsodium.23.dylib
error   00:44:26.485260+0200    kernel  System Policy: nix(3179) deny(1) file-read-data /nix/store/gbgryjc27mgd404q49ccjp6g5kf18c3v-libsodium-1.0.18/lib
default 00:44:26.488785+0200    ReportCrash ASI found [dyld] (sensitive) 'Library not loaded: /nix/store/gbgryjc27mgd404q49ccjp6g5kf18c3v-libsodium-1.0.18/lib/libsodium.23.dylib
  Referenced from: <no uuid> /nix/store/fzykws31cb5a88h3jkimzr944yaiqbyv-nix-2.15.0/bin/nix
  Reason: tried: '/nix/store/gbgryjc27mgd404q49ccjp6g5kf18c3v-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/System/Volumes/Preboot/Cryptexes/OS/nix/store/gbgryjc27mgd404q49ccjp6g5kf18c3v-libsodium-1.0.18/lib/libsodium.23.dylib' (no such file), '/nix/store/gbgryjc27mgd404q49ccjp6g5kf18c3v-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/usr/local/lib/libsodium.23.dylib' (no such file), '/usr/lib/libsodium.23.dylib' (no such file, not in dyld cache)'
error   00:44:26.849872+0200    osanalyticshelper   Sandbox extension token was missing.

I don't think there's much else to do here... launchd starts processes with permission context tied to the binary that is being started, and /bin/sh that we use doesn't have permissions to read removable media. I'll keep this VM image around, so if you need some additional info, I can gather it.

I found this article: https://chrispaynter.medium.com/what-to-do-when-your-macos-daemon-gets-blocked-by-tcc-dialogues-d3a1b991151f - it describes an approach where one binary is running both as daemon from root and as agent from user. It allows to request all necessary permissions from user as agent and they will be applied to the daemon as well. I think that instead of using /bin/sh and wait4path we should create a small binary wrapper wath4path-and-exec and install it this way. Then it will request all necessary permissions from user if needed.

andrewhamon commented 1 year ago

Seeing the same thing on Ventura 13.2.1 on ec2 (mac2.metal).

For any EC2 users out there, you can work around this by signing in over VNC and granting Full Disk Access permission to nix.

Here is the step-by-step:

Set a password sudo passwd ec2-user
Enable screen sharing sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate -configure -access -on -restart -agent -privs -all
Port forward VNC: ssh -L 5900:localhost:5900 ec2-user@<ip-address>
Open localhost:5900 in a VNC viewer. From a Mac, you can run open vnc://localhost:5900
Log in as ec2-user with the password you set earlier
On the remote Mac, go to System Settings > Privacy & Security > Full Disk Access
Enable nix

I'm not aware of a way to do this in a fully automated way, sadly. But the good news is that once you do this, you can create a custom AMI and use that for launching all your future instances.

toonn commented 1 year ago

@YorikSar, I assume this aligns with your findings? I.e. /bin/sh would inherit the Full Disk Access from Nix?

angerman commented 1 year ago

Sigh, this has been a bit annoying, lol... so here it goes. I'm using MDM to provision the Macs, and I'm using an external disk for nix (because it's bigger, cheaper, and reduces the wear on the internal small one). I could not get @YorikSar's /bin/sh FDA approach to work, neither giving /bin/sh outright FDA, nor with the copied executable. So what I ended doing was, use the following c-wrapper:

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>

int main() {
    pid_t pid;
    pid = fork();
    if (pid < 0) {
        fprintf(stderr, "Fork failed\n");
        return 1;
    }
    if (pid == 0) {
        char *args[] = {"/nix/var/nix/profiles/default/bin/nix-daemon", NULL};
        execvp(args[0], args);
        fprintf(stderr, "Failed to launch nix-daemon\n");
        return 1;
    } else {
        wait(NULL); 
        printf("done\n");
    }
    return 0;
}

and with (as root)

$CC nix-daemon-stub.c -o /var/root/nix-daemon

we now have a launcher that resides on the system disk (🤦). Changing /Library/LaunchDaemons/org.nixos.nix-daemon.plist to now look like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>EnvironmentVariables</key>
    <dict>
      <key>OBJC_DISABLE_INITIALIZE_FORK_SAFETY</key>
      <string>YES</string>
    </dict>
    <key>Label</key>
    <string>org.nixos.nix-daemon</string>
    <key>KeepAlive</key>
    <true/>
    <key>RunAtLoad</key>
    <true/>
    <key>ProgramArguments</key>
    <array>
      <string>/bin/sh</string>
      <string>-c</string>
      <string>/bin/wait4path /nix/var/nix/profiles/default/bin/nix-daemon &amp;&amp; exec /private/var/root/nix-daemon</string>
    </array>
    <key>StandardErrorPath</key>
    <string>/var/log/nix-daemon.log</string>
    <key>StandardOutPath</key>
    <string>/dev/null</string>
    <key>SoftResourceLimits</key>
    <dict>
      <key>NumberOfFiles</key>
      <integer>1048576</integer>
    </dict>
  </dict>
</plist>

and installing a TCC profile (I'm using mosyle), with Application Path: as /private/var/root/nix-daemon, and App Code Requirement: as identifier "nix-daemon", granting Full Disk Access, I can now launch nix-daemon just fine.

You can probably set this by hand as well to only that one executable and be done. It's still rather stupid to hack around like this.

angerman commented 1 year ago

echo "H4sICNHFB2UCA25peC1kYWVtb24A7d1tTBRHGAfw2T3QQ044Y8WztXi+VMFUoFirTURARFEREIi92jTLHbfAlXvr3R5CMXr9gK1pm0DSVtPYajStoa2pMdbUpIaDpi+xHxRbLTXRWmtSWmNCWmPEhNKZ2zlYljcT44em/18y2eeZnX12Z2f3vi2cu3frHxMhAqFE2sy0HYonpIVYWReZQ1sRbZJUlr+hcHtheSkZRSCTY3V6BFanstBWOcbxeboDeB6vaSR6HYrcqAwP09e7kaPW+1GTi9q6cSSsTSUpqIQcwXHrfbFGrVekybX1jLyeaUQ9qU52++XAGPVIrlqvSpNPfH3VQSXg8taOc31VvN42Ta4l6m6rJIW8O1xep+Ty1vjGmm+ubr65o2vo17U3WmddfmW+VFBaUlEZW78R66rbGniLPXOSVOtTRlzn2PUsmufDoBk/hTfNffN5g8q49Sy8npnvt+jqTTxPzY6wbp7h8eepXofbLgWbPA6fW/IrgXHrZWly7XVNpS1hxDyddsWuv2/DeRavZ9Tkk81Tfd+LN5ZsLly3MfZsRHTzjKibPuPwO6z9HVhArzyLj2NDjJGRr3gRzW3styaizlXka9sTUd+lnk5CWuk2kbYyMvIcIr+nCRPMYz89PnmM/iTarPx9zQwFA5lulyPT2eR28v3ZfL86l8RoE0jsntnFpZpnhkmj87Tw3xktVn8VnxO7XmG1KZ5VHTonbRVNQUX2ZKzNoOd3OdTjFvP65/l9S+d5D7/fj/Pc30VPPIUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAf9SgQei+XSq2DzQb2ge+Im3zrIaLRWZiMwp5/b3JpH85IW9bhbsrIwJ5mpDwhVU0rzIRPxuXTeOZJFxuIeQRln+XRGwGelxeAjGzPIn3s1aVRA5cM4RzIkaSxfJfDaRtDj2excbpQ8eVsXw5rfmqEC5fQLcpdExvcl4/69+bSGzpdFyuYLzZSUjlNcPMjweaC9pvl65tp/NoixikSwKvmTVNtBFx3iUWmzVxX8JwfF4TH9fErTxuSxCj8+jb0NEerWMk0f4yQiz9g4Oz2f1j8R0aW3j8N40X87iPxtk8vkXjPB7/SeNiGq/3BeqtNXaXW3ZOI5leV2Nmgz0Q3foDvhraHcx0yjX2kFvJdLi8bMcyp132+LyksFGuHjqyoM7ldlqrfR6/W1Zkon6fOpevr3bLvjVl36LShYy202vUfSxm37Ca6JFm/nWrwL42H4cn9/6+6wcAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAB2fh3/kv4tssvl0d+z/wfFuJvwcAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA8NDMWGDZupBsIzPyJEkKKvbqeqm6rl6qDdkDTrI1QFrVfqccCPhp4mxyO2kackgOl5d2klZeJ6ivUGN3ueneoJH2y41ydYOfZWaa1fgC9Sy2sHhHwKXILLPSzB9SgixOo/EOu0th1QWJxBNR8tRFq4QUWaqT7ezM8z12l5c8IdKTG8iNZn4dsWA3bUkJhGSF1b9bwIYlC+boPpanRHM6PkfN56r9Q3/iIEOX5+vyYl1eqcu363KHLq/T5V5dbqBtCm1TaTPSRqdCptEWR1s8bYkTjGGsRBrjnknRezZ6ncZYe82yx9ZPXbnYmqmrpa7TqMdCkqJd/oCrwU7HDrt31RShq9HML59ZpC6PWKvOXgzSMSwvIGIc4Ut3hDYbj6cT0uW3isR0f094XiwQvK7GZU677PF5yf4ma2vJJ3JSqPHN+E2HjpeXvBTu+PbRI8vuXlj50eyC/j2TFT3T99RbBy9sruhcdXZg98K5p/flGM7N6sz+4LAirD/qXzz/2O5PT9h83/xRE4n7qbu7Xiy1xQ/uPLnzlZbLZ94o2vvk0cn2P/va7RXdZ/2XW1YXixf3FJwq3JL73NVdKXtb9/2eVphx7IXPB3t+S2jouPle+v4lL6ZVhLvSrkgpH5715392KK3VOf/Ug55/sv3NpvfLunKvGF4+V5W+af07X19f+ti6zucPP5PSe/JOauesSw/7/HFLTnx5MFU43fLu9Y5fUgdXrAx+/3rByR96E7ek7or8deBn+gixh8mkPnKJAn9DAOD/7F9oFzzp8MgAAA==" \
 | base64 -d | gunzip > nix-daemon && chmod +x nix-daemon

for the lazy 🙄

yaoweizhen commented 1 year ago

I have no experience at all with sandboxing on MacOS, except as an end user. However, something that showed up when I searched was that there seems to be two different privileges, SystemPolicyAllFiles and SystemPolicyRemovableVolumes, and that despite the naming (and description) the first one does not overlap with the second, e.g. both are needed. Is nix-daemon asking for both?

In my System preferences, under Security and Integrity -> Integrity -> Files and Folders, Terminal is listed as having "Removable Volumes", while nix-daemon is listed with "Full Disk Access". (I'm using Swedish so the terminology might be a bit different.) Maybe nix-daemon inherits the Terminal-privileges when I execute it manually?

Battled for a weekend, finally seems to solve the problem. I searched either and the direction is right. Please try System preferences -> Security and Privacy -> Privacy -> Full Disk Access, here there is no nix-daemon on my list, so manually add it. Click the "+", it can't access /nix because of GUI selection, I navigated to my "nix store" volume to find the nix-daemon. However, it doesn't work as well. I believe it is a permission issue, and think about something missing, and I found a "sh" on the list, trying to check on it. Wait for a while, might be 1 or two minutes, the nix daemon on the Activity Monitor. Hopefully it is the solution.

arianvp commented 1 month ago

I found a hacky workaround for people on EC2:

Install https://tart.run on the instance, spawn a VM, then SSH into the VM and install Nix there. It's using Virtualization Framework so it's fast (native perfomance). This can all be done completely non-interactively.

NixOS / nix

Nix daemon blocked by sandbox on macOS Monterey (on external SSD) #6291