Open MichelV69 opened 8 months ago
MichelV69
commented
8 months ago Apologies, I forgot to include:
❯ ll /nix/var/nix/profiles/per-user
total 0
drwxr-xr-x@ 7 root nixbld 224B 6 Apr 01:21 root
I believe that is, in fact, mode 755 already
abathur
commented
8 months ago I imagine you're falling into what IIRC is an unresolved hole around trouble with external disks on macOS. Hopefully we'll finally be able to run at least one cause of this to ground.
It sounds like you've probably already seen it, but I'll go ahead and cross-link what I think is the most-commented issue on this for discoverability later (even though it also includes linux users running into the same message):
For now I'll just pepper you with questions:
diskutil info / and touch /?MichelV69
commented
8 months ago It sounds like you've probably already seen it, but I'll go ahead and cross-link what I think is the most-commented issue on this for discoverability later (even though it also includes linux users running into the same message):
Yes, but I haven't tried the chown $USER as a work around, just because "nothing is a permanent as a temporary fix" ;)
If you still have it, can you include the full install output under the output section (ideally with your install command visible)?
If you need, it's just ten minutes to uninstall / re-install.
What macOS version? (After looking up the max for a 2012 iMac I suspect it's Catalina, but if you're running anything earlier we may have a hole in the logic that decides whether we need a volume for Nix.)
... I was incorrect, I believe this was a 2018 machine. Sorry about that.
Machine info:
From the diskutil output, it looks like macOS itself is installed on one of the external drives? Are you using the internal drive used for anything?
Not yet. Is there a way to set that as the install target for Nix? I'd be more than happy to do that, beyond the part where it's a mechanical HDD.
Can you run and report output for
diskutil info /andtouch /?
Gladly. Files attached. diskutil.log
Oh. Well. I forgot about that:
❯ touch / > ~/Downloads/touch_root.log
touch: /: Operation not permitted
Continue to salt the discussion with a peppering of questions!
MichelV69
commented
8 months ago The plot thickens:
❯ sudo chown -R "$USER" /nix
chown: /nix/.Trashes: Operation not permitted
chown: /nix/.Trashes: Operation not permitted
- system: `"x86_64-darwin"`
- host os: `Darwin 22.6.0, macOS 10.16`
- multi-user?: `no`
- sandbox: `no`
- version: `nix-env (Nix) 2.21.1`
- channels(root): `"nixpkgs"`
- nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`
so, that foes function as a work-around for the problem
not elegant, but it is a work-around.
abathur
commented
8 months ago I'm not sure if it's related to the issue that brought you here, but nix-info shouldn't be reporting no for multi-user? on a ~correct install.
I imagine that's a sign that the daemon isn't running.
Occasionally it won't start right (most often if it's been disabled at some point and then removed without cleaning up the launchd state that tracks what is/isn't disabled, but I think there are some other cases). I guess the simple way to check is to run launchctl list | grep nixos and see if you have an org.nixos.nix-daemon and whether it lists a PID or not.
FWIW, a problem like this during install is exactly the kind of thing I'd be looking for in the full install output :)
MichelV69
commented
8 months ago I guess the simple way to check is to run launchctl list | grep nixos and see if you have an org.nixos.nix-daemon and whether it lists a PID or not.
❯ sudo launchctl list | grep nixos
Password:
- 0 org.nixos.darwin-store
- -6 org.nixos.nix-daemonFWIW, a problem like this during install is exactly the kind of thing I'd be looking for in the full install output :)
Ok, I've got something else I need to sort out first, and then I'll do the uninstall / re-install cycle and post the log.
MichelV69
commented
8 months ago given:
sh <(curl -L https://nixos.org/nix/install) --yes | tee ~/Downloads/nix_install.log
then ...
which gives ...
❯ nix-shell -p nix-info --run "nix-info -m"
error: could not set permissions on '/nix/var/nix/profiles/per-user' to 755: Operation not permittedLet me know if you need anything else, or if there is a way to move the Nix store partition to my internal mechanical.
MichelV69
commented
8 months ago I just found https://nixos.wiki/wiki/Storage_optimization#Moving_the_store but I'm not going to play with that until I'm sure you don't have any further questions with the current install
abathur
commented
8 months ago Did it emit any errors/warnings? (stderr wouldn't go through the pipe, and I don't see anything obvious in the log.)
Is the daemon service running? If not, can you run sudo launchctl print system, find the "disabled services" key, and see if the daemon's mentioned there? does sudo launchctl print system/org.nixos.nix-daemon show anything?
Can you see what these output?
sudo vsdbutil -c /
sudo vsdbutil -c /nix
diskutil info /nixI just found https://nixos.wiki/wiki/Storage_optimization#Moving_the_store but I'm not going to play with that until I'm sure you don't have any further questions with the current install
Those instructions probably assume linux. You can inevitably do something similar on macOS, but the read-only root in Catalina+ requires a bit of hoopjumping. The most ~humane way to do this is to set the envs below to tell it which disk to use (by default it'll glom on to whatever disk / is on) at install time. You can also pre-create a volume.
That said, I'm not sure it'll fix your issue (and having /nix on spinning rust will obviously be suboptimal), but it should clarify whether that's the source of the problem. (We know external drives are a common theme with people hitting this, and we know macOS treats the primary/internal volume a little differently, but we haven't shaken something preventive out of that knowledge.)
MichelV69
commented
8 months ago Did it emit any errors/warnings? (stderr wouldn't go through the pipe, and I don't see anything obvious in the log.)
All green. Nothing red / fail at any stage.
MichelV69
commented
8 months ago sudo launchctl print system/org.nixos.nix-daemon 2>&1 | tee ~/Downloads/launchctl_print.log
MichelV69
commented
8 months ago Is the daemon service running?
Yes, it is.
MichelV69
commented
8 months ago
❯ sudo vsdbutil -c / 2>&1 | tee ~/Downloads/vsdbutil_root.log
sudo vsdbutil -c /nix 2>&1 | tee ~/Downloads/vsdbutil_nix.log
diskutil info /nix 2>&1 | tee ~/Downloads/diskutil_nix.logbut it should clarify whether that's the source of the problem.
Okay. Going to have supper then I'll do the uninstall cycle and install after setting :
readonly NIX_VOLUME_USE_DISK="${NIX_VOLUME_USE_DISK:-$(root_disk_identifier)}"
NIX_VOLUME_USE_SPECIAL="${NIX_VOLUME_USE_SPECIAL:-}"
So, I likely did something wrong, but here we go:
given
❯ diskutil list
/dev/disk0 (internal, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *1.0 TB disk0
1: EFI EFI 209.7 MB disk0s1
2: Apple_APFS Container disk1 1000.0 GB disk0s2and
❯ export readonly NIX_VOLUME_USE_DISK="${NIX_VOLUME_USE_DISK:-/dev/disk0s2}"
❯ echo ${NIX_VOLUME_USE_DISK}
/dev/disk0s2
~~> Creating a Nix volume
---- sudo execution ------------------------------------------------------------
I am executing:
$ sudo /usr/sbin/diskutil apfs addVolume /dev/disk0s2 APFS Nix Store -nomount
to create a new APFS volume 'Nix Store' on /dev/disk0s2
Error starting APFS Volume creation: The given disk is not an APFS Container Reference (-69630)
---- oh no! --------------------------------------------------------------------
Oh no, something went wrong. If you can take all the output and open
an issue, we'd love to fix the problem so nobody else has this issue.
I'm guessing the target is just "disk0", but please confirm that.
MichelV69
commented
8 months ago NM. I realized what I was doing wrong. Should have used Disk1.
❯ diskutil list disk1
/dev/disk1 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +1000.0 GB disk1
Physical Store disk0s2
1: APFS Volume Nix Store 811.0 KB disk1s1
2: APFS Volume Preboot 24.6 KB disk1s2
3: APFS Volume Recovery 20.5 KB disk1s3
4: APFS Volume VM 2.1 GB disk1s4
5: APFS Volume Internal 1.0 MB disk1s5So I created the Vol from the GUI side of Disk Util as you see above. Ran the install and got:
~~> Fixing any leftover Nix volume state
Before I try to install, I'll check for any existing Nix volume config
and ask for your permission to remove it (so that the installer can
start fresh). I'll also ask for permission to fix any issues I spot.
---- warning! ------------------------------------------------------------------
Ignoring disk1s1 (96FE8B60-CE2C-41F1-BEE7-D40A51290926) because I am looking for:
disk=disk4 special=disk4sX uuid=anySo that's a no go.
MichelV69
commented
8 months ago Success!
Given
❯ export readonly NIX_VOLUME_USE_DISK="${NIX_VOLUME_USE_DISK:-disk1}"
❯ echo ${NIX_VOLUME_USE_DISK}
disk1and
❯ diskutil list disk1
/dev/disk1 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +1000.0 GB disk1
Physical Store disk0s2
1: APFS Volume Nix Store 1.4 GB disk1s1
2: APFS Volume Preboot 24.6 KB disk1s2
3: APFS Volume Recovery 20.5 KB disk1s3
4: APFS Volume VM 2.1 GB disk1s4
5: APFS Volume Internal 1.0 MB disk1s5then during install we see
~~> Creating a Nix volume
disk1s1 was already unmountedand in a new session we can sucessfully
❯ nix-shell -p nix-info --run "nix-info -m"
- system: `"x86_64-darwin"`
- host os: `Darwin 22.6.0, macOS 10.16`
- multi-user?: `yes`
- sandbox: `no`
- version: `nix-env (Nix) 2.21.1`
- channels(root): `"nixpkgs"`
- nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`
( took 19s)So it's a slow as a three turtles in a trench coat, but yes, it runs just fine.
❯ alias tick="ruby ~/Documents/Devel.Projects/lang.ruby/novel_writing.support_apps/afterlivingFG.clocktick.rbx"
❯ alias myRubocop="rubocop --parallel --autocorrect --color"
❯ nix-shell --run /bin/zsh
((package installation noise here ... I need to look up the option to just give me a progress bar))
❯ echo $NIX_STORE
/nix/store
❯ ruby -v
ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-darwin23]So it works just fine indeed if it's installed on the "internal" hard disk instead of the external boot disk.
MichelV69
commented
8 months ago Issue was closed by my cat, who was sure they were being helpful. :D
Re-opened by me, who is less convinced of that.
abathur
commented
8 months ago Thanks for confirming the internal disk works.
Since we're pretty confident it's the internal/external thing, here's my thesis:
If that thesis is right, I'm not aware of a way around that doesn't involve adding a Full Disk Access permission/exemption. If you switch back to the "normal" install (not overriding the disk), can you check Preferences > Security & Privacy > Privacy > Full Disk Access and see if org.nixos.nix-daemon shows up in the list of toggleable items? I suspect not, but if so, I'm curious what happens if you enable FDA for it and reboot.
If it isn't an option, you can probably get the same result by adding the FDA for /bin/sh (because the service invokes this first). I'll stop short of telling you to do that, since it comes with some risk (anything running under /bin/sh will have access to the locations Full Disk Access).
Unless/until we find some way around this that doesn't involve FDA, there isn't anything we can do to pre-fix this in the installer (macOS won't let us add this permission), but we could/should probably try to detect external volumes and either throw a fit or notify the user that they'll need to take remedial steps.
MichelV69
commented
8 months ago If I have time tomorrow, I'll do that test for you.
Now, I can tell you that it's possible to run an exe from an external, because that's where I have Chrome installed and a couple of other self-updating apps that launch and run background tasks. So it's going to be a secret sauce issue to figure out how the installer should be launching to let this work right.
MichelV69
commented
7 months ago ok. finally able to work on this. Just getting my environment uninstalled
MichelV69
commented
7 months ago okay. 2m30s, and done a "standard" install 100% to the external drive, which results in the problem as expected:
❯ nix-shell -p nix-info --run "nix-info -m"
error: could not set permissions on '/nix/var/nix/profiles/per-user' to 755: Operation not permittedChecking Full Disk Access shows:
So, it's the daemon isn't present, as you expected. I've now just tried to add it, however, I'm having trouble finding the full path to the silly thing. Hunting for it now
MichelV69
commented
7 months ago Soooooo, no I can't. It's not available as an option. It's greyed out / unchoosable. Which is unfortunate.
MichelV69
commented
7 months ago There are days I'm frightened that they let me have a driver's licence.
I just realized that the "nix" in the list there didn't stand for *nix, like I'm used to thinking. It's literally the entire Nix-Shell environment.
... and now it works just fine:
❯ nix-shell -p nix-info --run "nix-info -m"
- system: `"x86_64-darwin"`
- host os: `Darwin 22.6.0, macOS 10.16`
- multi-user?: `yes`
- sandbox: `no`
- version: `nix-env (Nix) 2.21.1`
- channels(root): `"nixpkgs"`
- nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`I'm going to keep experimenting here, and I'll let you know if anything crops up.
abathur
commented
7 months ago That's a good sign. In some other contexts people haven't had luck with exempting the nix executable, or i would have explicitly suggested it.
PlumpMath
commented
4 months ago Is there a way to install macOS and the Nix daemon on the Mac’s original hard drive as usual, but install all additional Nix stores on an external hard drive? On Linux systems other than NixOS, I’ve been solving this issue by moving the Nix store to another location and running the following script at every reboot:
mkdir -pv /nix
mount --bind /home/user/new-nix-store /nixI hope that a similar solution could resolve the multi-user issue on macOS. To achieve this, we might need a way to bypass mounting /nix on a new volume of APFS. Does anyone know how to do this?
If such an option is added to the Nix installer for macOS, it would perfectly solve the issue.
PlumpMath
commented
2 months ago I solved this issue by creating an image of the volume containing the Nix store right after the initial Nix installation, then transferring it to an external hard drive using Carbon Copy Cloner, formatted as APFS. (When formatting APFS, I chose the regular APFS option, not case-sensitive.) After that, I followed the steps you explained by replacing the entries in /etc/fstab and /Library/Launch... with the new APFS-formatted volume. Then, after rebooting and enabling the permissions for Nix and various Nix-related programs that run in the terminal (like nix sh), the issue with the daemon, file permissions, and Nix breaking was completely resolved.
During the resolution process, especially when building the system profile, it’s crucial to preemptively enable all permissions for Nix and Nix-related sh and bash processes to prevent Nix from breaking.
https://discourse.nixos.org/t/how-to-move-nix-store-to-external-drive-on-macos/19592/3
Thank you for writing this guide.
Platform
Additional information
iMac 21" Intel 2012. Reinstalled the system today, wiped all drives. Installed XCode, Hyper, VSCode, then ran the Nix "Multi-user installation" per the website with the
--yesflag.Opened new terminal, ran
nix-shell -p nix-info --run "nix-info -m"and it errors out witherror: could not set permissions on '/nix/var/nix/profiles/per-user' to 755: Operation not permittedReviewed other issues such as #6667 with similar error message, but solutions did not seem to apply.
Output
Output
```log **Current state of machine, after most recent re-install** ❯ stat /nix /etc /etc/{fstab,synthetic.conf,.fstab.swp} ~/.nix-profile 16777252 2 drwxr-xr-x 5 root nixbld 0 160 "Apr 6 01:22:44 2024" "Apr 6 01:20:43 2024" "Apr 6 01:20:43 2024" "Apr 6 01:20:15 2024" 4096 0 0 /nix 16777232 1152921500312436257 lrwxr-xr-x 1 root wheel 0 11 "Mar 17 02:25:17 2024" "Mar 17 02:25:17 2024" "Mar 17 02:25:17 2024" "Mar 17 02:25:17 2024" 4096 0 0x88000 /etc 16777232 909360 -rw-r--r-- 1 root wheel 0 206 "Apr 6 01:31:16 2024" "Apr 6 01:20:16 2024" "Apr 6 01:20:16 2024" "Apr 6 00:34:03 2024" 4096 8 0 /etc/fstab 16777232 939784 -rw-r--r-- 1 root wheel 0 4 "Apr 6 01:20:15 2024" "Apr 6 01:20:15 2024" "Apr 6 01:20:15 2024" "Apr 6 01:20:15 2024" 4096 8 0 /etc/synthetic.conf stat: /etc/.fstab.swp: stat: No such file or directory stat: /Users/mvaillancourt/.nix-profile: stat: No such file or directory ❯ cat /etc/fstab # # Warning - this file should only be modified with vifs(8) # # Failure to do so is unsupported and may be destructive. # UUID=83301658-D79F-4F50-A0E4-A99099F42D78 /nix apfs rw,noauto,nobrowse,suid,owners ❯ diskutil list /dev/disk0 (internal, physical): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *1.0 TB disk0 1: EFI EFI 209.7 MB disk0s1 2: Apple_APFS Container disk1 1000.0 GB disk0s2 /dev/disk1 (synthesized): #: TYPE NAME SIZE IDENTIFIER 0: APFS Container Scheme - +1000.0 GB disk1 Physical Store disk0s2 1: APFS Volume Preboot 24.6 KB disk1s2 2: APFS Volume Recovery 20.5 KB disk1s3 3: APFS Volume VM 2.1 GB disk1s4 4: APFS Volume Internal 991.2 KB disk1s5 /dev/disk2 (external, physical): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *1.0 TB disk2 1: EFI EFI 209.7 MB disk2s1 2: Apple_APFS Container disk9 1000.0 GB disk2s2 /dev/disk3 (external, physical): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *1.0 TB disk3 1: Microsoft Reserved 134.2 MB disk3s1 2: Apple_APFS Container disk4 1.0 TB disk3s2 /dev/disk4 (synthesized): #: TYPE NAME SIZE IDENTIFIER 0: APFS Container Scheme - +1.0 TB disk4 Physical Store disk3s2 1: APFS Volume Crucial X6 - Data 162.7 GB disk4s1 2: APFS Volume Preboot 2.1 GB disk4s2 3: APFS Volume Recovery 1.2 GB disk4s3 4: APFS Volume Crucial X6 9.3 GB disk4s4 5: APFS Snapshot com.apple.os.update-... 9.3 GB disk4s4s1 6: APFS Volume VM 1.1 MB disk4s6 7: APFS Volume Nix Store 421.4 MB disk4s7 /dev/disk5 (external, physical): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *64.0 GB disk5 1: EFI EFI 209.7 MB disk5s1 2: Apple_RAID 63.7 GB disk5s2 3: Apple_Boot Boot OS X 134.2 MB disk5s3 /dev/disk6 (external, physical): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *64.0 GB disk6 1: EFI EFI 209.7 MB disk6s1 2: Apple_RAID 63.7 GB disk6s2 3: Apple_Boot Boot OS X 134.2 MB disk6s3 /dev/disk7 (external, virtual): #: TYPE NAME SIZE IDENTIFIER 0: Apple_APFS +127.4 GB disk7 /dev/disk8 (synthesized): #: TYPE NAME SIZE IDENTIFIER 0: APFS Container Scheme - +127.4 GB disk8 Physical Store disk7 1: APFS Volume SSD Raid 7.2 GB disk8s1 /dev/disk9 (synthesized): #: TYPE NAME SIZE IDENTIFIER 0: APFS Container Scheme - +1000.0 GB disk9 Physical Store disk2s2 1: APFS Volume OrangeBox 920.6 GB disk9s2 ❯ id uid=501(mvaillancourt) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),701(com.apple.sharepoint.group.1),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),400(com.apple.access_remote_ae) ``` Have repeatedly uninstalled per https://nixos.org/manual/nix/stable/installation/uninstall.html#macos ensuring that all steps are followed, and even rebooted twice for good measure. All install attempts result in the same behaviour.Priorities
Add :+1: to issues you find important.