Closed svenssonaxel closed 1 month ago
grahamc
commented
11 months ago This is super interesting. We test regularly on a fleet of macs, but they're in a rack and don't use external storage. Thanks for such a good log / record of what you're seeing, we'll hunt this down.
grahamc
commented
11 months ago Notes from a mac on AWS, running macOS Sonoma 14.1 on an mac2-m2.metal, which I'd expect to exhibit similar symptoms.
ec2-user@ip-172-31-40-68 ~ % mount
/dev/disk5s2s1 on / (apfs, sealed, local, read-only, journaled)
devfs on /dev (devfs, local, nobrowse)
/dev/disk5s5 on /System/Volumes/VM (apfs, local, noexec, journaled, nobrowse)
/dev/disk5s3 on /System/Volumes/Preboot (apfs, local, journaled, nobrowse)
/dev/disk1s2 on /System/Volumes/xarts (apfs, local, noexec, journaled, nobrowse)
/dev/disk1s1 on /System/Volumes/iSCPreboot (apfs, local, journaled, nobrowse)
/dev/disk1s3 on /System/Volumes/Hardware (apfs, local, journaled, nobrowse)
/dev/disk5s1 on /System/Volumes/Data (apfs, local, journaled, nobrowse)
map auto_home on /System/Volumes/Data/home (autofs, automounted, nobrowse)
/dev/disk3s4 on /private/tmp/tmp-mount-qYjtWH (apfs, local, journaled, nobrowse)ec2-user@ip-172-31-40-68 ~ % diskutil list
/dev/disk0 (internal, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *251.0 GB disk0
1: Apple_APFS_ISC Container disk1 524.3 MB disk0s1
2: Apple_APFS Container disk3 245.1 GB disk0s2
3: Apple_APFS_Recovery Container disk2 5.4 GB disk0s3
/dev/disk3 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +245.1 GB disk3
Physical Store disk0s2
1: APFS Volume InternalDisk 9.9 GB disk3s1
2: APFS Volume Preboot 5.1 GB disk3s2
3: APFS Volume Recovery 870.3 MB disk3s3
4: APFS Volume Data 560.0 MB disk3s5
5: APFS Volume VM 20.5 KB disk3s6
/dev/disk4 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *107.4 GB disk4
1: EFI EFI 209.7 MB disk4s1
2: Apple_APFS Container disk5 107.2 GB disk4s2
/dev/disk5 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +107.2 GB disk5
Physical Store disk4s2
1: APFS Volume Macintosh HD - Data 4.8 GB disk5s1
2: APFS Volume Macintosh HD 9.9 GB disk5s2
3: APFS Snapshot com.apple.os.update-... 9.9 GB disk5s2s1
4: APFS Volume Preboot 5.6 GB disk5s3
5: APFS Volume Recovery 820.5 MB disk5s4
6: APFS Volume VM 20.5 KB disk5s5
svenssonaxel
commented
11 months ago Not that I've used other macs on AWS withtout this problem, so if you have trouble reproducing, make sure to use the exact same instance type and OS version.
grahamc
commented
11 months ago I could replicate this with the host above. Here's the relevant part of the plan:
{
"version": "0.15.1",
"actions": [
{
"action": {
"action": "create_apfs_volume",
"disk": "disk5",
"name": "Nix Store",
"case_sensitive": false,
"encrypt": false,
"create_or_append_synthetic_conf": {
"action": {
"path": "/etc/synthetic.conf",
"user": null,
"group": null,
"mode": null,
"buf": "nix\n",
"position": "End"
},
"state": "Uncompleted"
},
"create_synthetic_objects": {
"action": null,
"state": "Uncompleted"
},
"unmount_volume": {
"action": {
"disk": "disk5",
"name": "Nix Store"
},
"state": "Uncompleted"
},
"create_volume": {
"action": {
"disk": "disk5",
"name": "Nix Store",
"case_sensitive": false
},
"state": "Uncompleted"
},
"create_fstab_entry": {
"action": {
"apfs_volume_label": "Nix Store",
"existing_entry": "None"
},
"state": "Uncompleted"
},
"encrypt_volume": null,
"setup_volume_daemon": {
"action": {
"path": "/Library/LaunchDaemons/org.nixos.darwin-store.plist",
"apfs_volume_label": "Nix Store",
"mount_service_label": "org.nixos.darwin-store",
"mount_point": "/nix",
"encrypt": false,
"needs_bootout": false
},
"state": "Uncompleted"
},
...and I got very similar output:
Proceed? ([Y]es/[n]o/[e]xplain): y
INFO Step: Create an APFS volume `Nix Store` for Nix on `disk5` and add it to `/etc/fstab` mounting on `/nix`
INFO Step: Provision Nix
INFO Step: Create build users (UID 300-332) and group (GID 30000)
INFO Step: Configure Time Machine exclusions
INFO Step: Configure Nix
INFO Step: Configuring zsh to support using Nix in non-interactive shells
INFO Step: Create a `launchctl` plist to put Nix into your PATH
INFO Step: Configure Nix daemon related settings with launchctl
INFO Step: Remove directory `/nix/temp-install-dir`
Nix was installed successfully!
To get started using Nix, open a new shell or run `. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh`
ec2-user@ip-172-31-40-68 ~ % . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
ec2-user@ip-172-31-40-68 ~ % nix repl
error: could not set permissions on '/nix/var/nix/profiles/per-user' to 755: Operation not permitted
ec2-user@ip-172-31-40-68 ~ % tail /var/log/system.log
Nov 25 00:04:31 ip-172-31-40-68 syslogd[98]: ASL Sender Statistics
ec2-user@ip-172-31-40-68 ~ % launchctl list | grep nix-daemon
- -6 org.nixos.nix-daemon
ec2-user@ip-172-31-40-68 ~ % sudo launchctl start org.nixos.nix-daemon
ec2-user@ip-172-31-40-68 ~ % launchctl list | grep nix-daemon
- -6 org.nixos.nix-daemon
ec2-user@ip-172-31-40-68 ~ % tail /var/log/system.log
Nov 25 00:04:31 ip-172-31-40-68 syslogd[98]: ASL Sender Statistics
ec2-user@ip-172-31-40-68 ~ % tail /var/log/nix-daemon.log
Reason: tried: '/nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/System/Volumes/Preboot/Cryptexes/OS/nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib' (no such file), '/nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/usr/local/lib/libsodium.23.dylib' (no such file), '/usr/lib/libsodium.23.dylib' (no such file, not in dyld cache)
dyld[1417]: Library not loaded: /nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib
Referenced from: <no uuid> /nix/store/0pbq6wzr2f1jgpn5212knyxpwmkjgjah-nix-2.18.1/bin/nix
Reason: tried: '/nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/System/Volumes/Preboot/Cryptexes/OS/nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib' (no such file), '/nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/usr/local/lib/libsodium.23.dylib' (no such file), '/usr/lib/libsodium.23.dylib' (no such file, not in dyld cache)
dyld[1420]: Library not loaded: /nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib
Referenced from: <no uuid> /nix/store/0pbq6wzr2f1jgpn5212knyxpwmkjgjah-nix-2.18.1/bin/nix
Reason: tried: '/nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/System/Volumes/Preboot/Cryptexes/OS/nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib' (no such file), '/nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/usr/local/lib/libsodium.23.dylib' (no such file), '/usr/lib/libsodium.23.dylib' (no such file, not in dyld cache)
dyld[1430]: Library not loaded: /nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib
Referenced from: <no uuid> /nix/store/0pbq6wzr2f1jgpn5212knyxpwmkjgjah-nix-2.18.1/bin/nix
Reason: tried: '/nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/System/Volumes/Preboot/Cryptexes/OS/nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib' (no such file), '/nix/store/gky12ai26saxyvki60g0zld0sank42c9-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/usr/local/lib/libsodium.23.dylib' (no such file), '/usr/lib/libsodium.23.dylib' (no such file, not in dyld cache)I have solved this, but it'll take time to get it integrated into the installation flow. I've created a signed program that shims access to the Nix daemon. This gets us half way there. It has a caveat, though: The machine must grant Full Disk Access to that program.
This can be done in two ways:
This is basically the only way, by design by Apple. Their security is good! :).
Next steps:
Big thanks to @angerman whose comments on that linked issue were right on point, and got me a lot of the way through.
svenssonaxel
commented
11 months ago @grahamc Triggered by a comment in the linked issue I tried to run the daemon manually, not as a service, in a separate terminal. This workaround appeared to be successful. Does this change your assessment in any way, as to whether the shim is necessary?
angerman
commented
11 months ago If you run nixin Terminal.app, and Terminal.app has FDA, so will nix (inherit).
What you can do, though that's really a bit, uh, hacky:
😒 I don't want to go there again.
svenssonaxel
commented
11 months ago @angerman This was done with no graphical session. I just did ssh, sudo, nix-daemon. If that gives FDA then I don't understand why a special shim is necessary, and if that does not give FDA then I don't understand what prevents nix-daemon started as a service from working and why a shim is necessary. I'm probably wrong, just wanted to point out the manual workaround if that could give @grahamc any ideas.
grahamc
commented
11 months ago Unfortunately the shim is still likely to be necessary in almost all cases. The behavior of the sandbox is a bit mysterious, and different behaviors can make it be a non-issue... sometimes. The shim isn't very complicated, though, which is good news.
angerman
commented
11 months ago @svenssonaxel you may be able to verify three things: 1.) check if SIP is enabled (csrutil status, I think) 2.) if /nix is indeed on an external disk. (mount and diskutil should be able to verify) 3.) log into the graphical session and check which applications have FullDiskAccess in your Settings.app
Iirc if ssh-keygen-wrapper (or whatever that thing is called) has FDA, so will your ssh session as well.
By default on a blank macOS install nothing has FDA permissions.
svenssonaxel
commented
11 months ago @svenssonaxel you may be able to verify three things:
@angerman Sorry, I don't have access to such a machine currently.
kbob
commented
2 months ago I'm trying to install nix for the first time, so please excuse my ignorance.
I hit the same error tonight on an Intel iMac Pro booted from an external drive, MacOS 14.6.1 Sonoma. Same errors in /var/log/nix-daemon.log. I used the command-line installer. The install script reported success, but the self-test subcommand (and "nix run nixpkgs#hello") reported
error: could not set permissions on '/nix/var/nix/profiles/per-user' to 755: Operation not permittedI eventually traced it down to the same errors in /var/log/nix-daemon:
dyld[29325]: Library not loaded: /nix/store/hgs3ai43fdac79v6dvc2k8zz38jhqfzj-libsodium-1.0.18/lib/libsodium.23.dylib
Referenced from: <no uuid> /nix/store/51zkf3552d20gm6jswz3xs1yipdgksbg-nix-2.23.3/bin/nix
Reason: tried: '/nix/store/hgs3ai43fdac79v6dvc2k8zz38jhqfzj-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/System/Volumes/Preboot/Cryptexes/OS/nix/store/hgs3ai43fdac79v6dvc2k8zz38jhqfzj-libsodium-1.0.18/lib/libsodium.23.dylib' (no such file), '/nix/store/hgs3ai43fdac79v6dvc2k8zz38jhqfzj-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/usr/local/lib/libsodium.23.dylib' (no such file), '/usr/lib/libsodium.23.dylib' (no such file, not in dyld cache)Is this problem fixed in the graphical installer, and is that why it's a signed app?
Thanks.
grahamc
commented
1 month ago Hey folks, this problem (which is specific to macOS on EC2) has been solved. See: https://determinate.systems/posts/unattended-nix-install-macos-aws-ec2/
If the daemon startup fails, the installation is still reported as successful. While this might be good for statistics, it might be better to check that the daemon has started and report failure otherwise, so that the issues can be addressed.
In this particular case, the failure looks similar to https://github.com/NixOS/nix/issues/6291 after running nix-installer on an AWS mac2-m2pro.metal.