amckinlay commented 4 years ago

I have noticed that every once and a while /Users/andrewmckinlay/.nix-profile/bin and /nix/var/nix/profiles/default/bin disappear from my $PATH. The last time this happened, the hook nix installed was missing from /etc/zshrc.

Copying the following hook (copied from /etc/bashrc) to /etc/zshrc and restarting the shell fixed everything again:

# Nix
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
  . '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End Nix

My concern is that macOS updates will periodically overwrite /etc/.zshrc (and /etc/.bashrc), requiring effort on my part to reinstall nix. For example, one macOS update overwrote the entire file to fix this misspelling:

andrewmckinlay@imac ~ % diff /etc/zshrc.backup-before-nix /etc/zshrc
3c3
< # Setup user specific overrides for this in ~/.zhsrc. See zshbuiltins(1)
---
> # Setup user specific overrides for this in ~/.zshrc. See zshbuiltins(1)

Both files are read-only anyway, should they really be written to by the nix installer?

lilyball commented 4 years ago

If they keep updating these files, perhaps we should file an Apple Feedback report asking them to add /etc/bashrc.d/ and /etc/zshrc.d folders whose contents get sourced by the default scripts? This way we can add configuration there that won't get overridden.

Beyond that, the only automatic solution that comes to mind would be to have nix-daemon check these files on launch and re-add the Nix block if it's missing.

ptitvert commented 4 years ago

I have done a few days before the upgrade to Catalina 10.15.6... and today I wanted to install something to discover that my nix installation was gone! Well after some research, I found that the /nix was still there, but everything else was gone... no nix-env command nothing! So after cursing a lot... I search until I found this thread... I would like also that this problem to solved... For the moment, I have put on my 2 users the following lines in the ~/.profile

# Nix
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
  . '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End Nix

So I don't care if the /etc/zshrc is or not overwritten in the future... I agree this is a bad solution, but at least for the time being my workaround...

lilyball commented 4 years ago

I filed feedback for Apple (FB8181728). In the meantime, nix-daemon will need to learn how to modify the files appropriately on launch.

abathur commented 4 years ago

~~The zsh case seems to have been fixed (but not released yet) in the course of #3456. I haven't observed this same issue with bash shell files.~~

Edit: this had its own problems and was reverted.

nixos-discourse commented 3 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/here-are-the-etc-bashrc-amendments-performed-by-the-installer-on-macos/11360/1

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

elliot-u410 commented 3 years ago

I'm still seeing this issue as of Nix 2.3.15 and Big Sur 11.5.2. The workaround is to re-add the following to /etc/zshrc:

# Nix
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
  source '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End Nix

abathur commented 3 years ago

I'm still seeing this issue as of Nix 2.3.15 and Big Sur 11.5.2.

Edit: I see in matrix that #3608 caused its own problems documented in #4169, so I guess a release would actually be a step in the wrong direction atm... Striking through the parts that may not actually be much help...

FWIW this is roughly expected:

~The fix for the ZSH case was merged in #3608, but there hasn't been a release from master since then (and it doesn't look like anyone's backported it). I'm not really in the loop, but at least per https://discourse.nixos.org/t/nix-release-schedule-and-roadmap/14204 there should be a release from master coming fairly soon?~
Even after a new release is cut from master, I expect you'll continue to see this behavior on an old install until you manually migrate or manually remove Nix and reinstall it.

~I would try manually migrating. I think you'd just need to create /etc/zshenv if it doesn't exist, move your source line there, and verify it works.~

~Also: brownie points if you (or maybe any zsh user who hasn't taken 11.5.2 yet?) remember to report back here or in #3608 if this migration helps you dodge this problem.~

ggPeti commented 3 years ago

can't we just patch zsh to use different rcfile paths? users can then set the patched zsh as their shell with nix-darwin

abathur commented 3 years ago

@ggPeti I assume it's ideal if it still works without Nix-darwin, but tbh I have no clue how big the set of (Nix + zsh - nix-darwin) uses is.

4169 seems to have a more robust discussion on ~zsh-specific fixes, and I'm trying to fish around for someone to add it to the 2.4 milestone; if I find someone, that might be a good place to pull on the thread...

abathur commented 3 years ago

4169 got closed without dealing with this part of the problem.

abathur commented 3 years ago

I recently stumbled on a reference to someone saying they'd taken an update and had the update leave their modified items in place, and add the replaced items to the Relocated Items directory.

I have a spare laptop on the beta release channel and wanted to try this out, so I took an update on it this afternoon to 11.5 and noticed two things:

It did not displace /etc/bashrc on this update--it just placed the "official" bashrc at Relocated\ Items/Configuration/private/etc/bashrc.system_default.
It did, however, still overwrite /etc/zshrc.

I'm not sure if this means they're treating zsh init files differently since it's the default shell, or if it means they inadvertently left zshrc out of the list of files to respect instead of replace. Something to watch closely in the next few updates, I guess.

abathur commented 3 years ago

I recently stumbled on a reference to someone saying they'd taken an update and had the update leave their modified items in place, and add the replaced items to the Relocated Items directory.

I have a spare laptop on the beta release channel and wanted to try this out, so I took an update on it this afternoon to 11.5 and noticed two things:

It did not displace /etc/bashrc on this update--it just placed the "official" bashrc at Relocated\ Items/Configuration/private/etc/bashrc.system_default.

It did, however, still overwrite /etc/zshrc.

I'm not sure if this means they're treating zsh init files differently since it's the default shell, or if it means they inadvertently left zshrc out of the list of files to respect instead of replace. Something to watch closely in the next few updates, I guess.

See also: https://developer.apple.com/forums/thread/670671

I tried this with an update from 12.0 beta to 12.0.1 and, once again, zshrc was overwritten.

nixos-discourse commented 3 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/macos-update-and-nix/15779/2

nixos-discourse commented 2 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-commands-missing-after-macos-12-1-version-upgrade/16679/5

ctheune commented 2 years ago

And of course this happened again on 12.2. (@frlan)

amckinlay commented 2 years ago

Please deprecate nix on macOS.

ctheune commented 2 years ago

... please don't?!?

amckinlay commented 2 years ago

Every macOS update breaks nix. It's clearly not ready for macOS.

asivitz commented 2 years ago

When trying to debug this problem for myself, I think I was misled by the current documentation. For example, this discussion of environment variables

https://nixos.org/manual/nix/stable/installation/env-variables.html

assumes a single-user installation. (I think?) And because I don't have prefix/etc/profile.d/nix.sh I thought my installation might be broken. Am I right in thinking there should be separate sections for single and multi user installations and the multi-user section should mention nix-daemon.sh instead of nix.sh?

abathur commented 2 years ago

assumes a single-user installation... Am I right in thinking there should be separate sections for single and multi user installations and the multi-user section should mention nix-daemon.sh instead of nix.sh?

Good catch. I think so. If none of the differences between the two scripts seem to fall within the scope of the current documentation, it might suffice to just say that each mode has a distinct script and list the path for each?

asivitz commented 2 years ago

That sounds reasonable to me

thiagowfx commented 2 years ago

Happened to me today, macOS Monterey 12.2 -> 12.3 update.

Fix that worked for me:

# 1. Remove backup files
sudo rm /etc/{bashrc,zshrc}.backup-before-nix

# 2. Run installation script again
sh <(curl -L https://nixos.org/nix/install)

# Doing 2. before 1. makes the installation script fail.

abathur commented 2 years ago

@thiagowfx You don't have to reinstall. As noted multiple times in the thread, you just need to re-add the profile/rc hook to /etc/bashrc or /etc/zshrc.

iFreilicht commented 2 years ago

And because I don't have prefix/etc/profile.d/nix.sh I thought my installation might be broken.

Same for me. I "fixed" it by running

/nix/store/*-nix-2.*/bin/nix-env -iA nixpkgs.nix

Which creates all the same links in your per-user profile, but of course, that's only a viable workaround for a single user on a multi-user install.

nixos-discourse commented 2 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/fixing-your-install-after-osx-upgrade/19339/6

Gau-thier commented 2 years ago

Same issue on my side, while updating from Monterey 12.1 to Monterey 12.5 on a Apple M1 Pro... Fixed by adding the following lines on /etc/zshrc

# Nix
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
  . '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End Nix

somebodyLi commented 2 years ago

Fixed by adding the following lines on /etc/zshrc

how can i do if i have remove the pre-installed version, Now it can't install successful ........

abathur commented 2 years ago

Fixed by adding the following lines on /etc/zshrc

how can i do if i have remove the pre-installed version, Now it can't install successful ........

@somebodyLi The instructions didn't say to remove this system file, so I don't know why you removed it.

If you still have /etc/zshrc.backup-before-nix, all you need to do is sudo mv /etc/zshrc.backup-before-nix /etc/zshrc (as the instructions say).
If you don't have the file, you're going to have to find a replacement. Restore it from backups if you have them. Otherwise, you could re-create it using the version in Apple's org on github: https://github.com/apple-oss-distributions/zsh/blob/zsh-92/zshrc. I have no idea if this file is up-to-date.

nixos-discourse commented 2 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-install-breaks-after-every-macos-monterey-update/21353/4

erikkrieg commented 2 years ago

I have encountered what I think is another flavour of this issue (https://github.com/NixOS/nix/issues/7106), except this time I lost /etc/fstab on update and my nix volume was modified during the upgrade to mount /Volumes/Nix\ Store instead of /nix. This breaks all the symlinks. Trying to figure out what how to correctly change the mount point back to /nix (just started looking because I just figured out roughly what was going on).

Edit: This was upgrading 12.5.1 to 12.6, btw

Edit 2: I was able to resolve the issue and think that it might have coincidentally lined up with an OS upgrade. The /etc files might have been removed by multiple runs of the install script but there was no indication of an issue until a system restart, which changed the volume mount point. That is the new theory, at least.

NilsIrl commented 2 years ago

For future reference this is what /etc/fstab should look like:

#
# Warning - this file should only be modified with vifs(8)
#
# Failure to do so is unsupported and may be destructive.
#
UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX /nix apfs rw,noauto,nobrowse,suid,owners

With the appropriate UUID.

Edit 2: I was able to resolve the issue and think that it might have coincidentally lined up with an OS upgrade. The /etc files might have been removed by multiple runs of the install script but there was no indication of an issue until a system restart, which changed the volume mount point. That is the new theory, at least.

FWIW I'm currently on 12.6 and had no issue whatsover when upgrading to it (other than the usual /etc/zshrc).

erikkrieg commented 2 years ago

Yeah, I used https://gist.github.com/meeech/0b97a86f235d10bc4e2a1116eec38e7e#check-you-have-nix to help me get back to a working state. It was both /etc/synthetic.confg and /etc/fstab that I needed to re-create.

alphatownsman commented 1 year ago

I just upgraded to macOS 13.1 (22C65), it seems /nix survived.

➜  ~ diskutil list | grep Nix
   7:                APFS Volume Nix Store               1.4 GB     disk3s7
➜  ~ diskutil info disk3s7 | grep Mount
   Mounted:                   Yes
   Mount Point:               /nix

Hence the only restoration really need is ~/.zshrc or /etc/zshrc

. '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'

FloThinksPi commented 1 year ago

@alphatownsman for me nix always survived but the /etc/zshrc is overwritten in every update needing manual adding the shell hook back into this file to make it work again.

As an other issue noted https://github.com/NixOS/nix/issues/3456 we should use /etc/zshenv which is global and loaded by default BUT not overwritten by an OS update since it does not exist by default.

Tested it and that way the shell hook survived the last macos update. No action was needed (adding sth back to /etc/zshrc)

IMHO we should change the installer script to use /etc/zshenv instead of /etc/zshrc for mac

abathur commented 1 year ago

@FloThinksPi we tried zshenv but reverted because it causes its own problem. See https://github.com/NixOS/nix/issues/4169

Edit: but it's a fine approach for individuals who want their system binaries to have priority. This isn't the normal case, but I know there are at least a few people out there who prefer that behavior.

FloThinksPi commented 1 year ago

@abathur Dang! Yea i also dont want that prio of binaries. Hm there must be a way to have the shell hook somewhere that does not get overwritten, brew e.g. puts it in the users shell env by printing a snipped people have to add themselfs. Maybe we should just print the snipped in the installer too and let people figure out where to put the hook. Thats also how pyenv and other tools which mess with the shell do it.

Docker does this also to the users ~/.zshrc but instead of printing it, inserts it on its own which i feel like many people would not like since i personally dont like a tool altering my versioned dotfiles.

source /Users/<username>/.docker/init-zsh.sh || true # Added by Docker Desktop

Anyway that way people would not expect the nix installer to manage that part and we could close the issue that way.

domenkozar commented 1 year ago

The issue is that we can't use /etc/zshenv because the $PATH we set gets overriden by /etc/zprofile.

Instead we use /etc/zshrc, which gets overriden by macOS on upgrades.

There's a detailed explanation how the startup of zsh works: https://gist.github.com/Linerre/f11ad4a6a934dcf01ee8415c9457e7b2

The only solution I can think of is:

prepend /nix/var/nix/profiles/default/bin to /etc/paths
when invoking nix check for __ETC_PROFILE_NIX_SOURCED=1 and if absent, tell the user to run a command that will populate /etc/zshrc.

Nix will still break, but the fix for the user is provided in a nice way.

abathur commented 1 year ago

I communicated this issue upstream to our devrel this past fall (it was passed along, but I haven't heard anything else).

One ~alternative solution I've seen suggested is either updating nix-daemon or installing a separate daemon that will fix-up if it goes missing.

Another is just caving and using ~/.zshrc.

I don't remember for sure, but we might also be able to try to pre-empt path_helper (which is what screws up the PATH if we use zshenv, and what processes /etc/paths and /etc/paths.d/*) by declaring a function in zshenv with its absolute path (/usr/libexec/path_helper), and then implementing it as a wrapper there that can ~fix the PATH it generates. (But I'm just assuming this is valid in zsh; function names with slashes are at least valid in bash).

domenkozar commented 1 year ago

~/.zshrc means that it's very hard to support multi-user installs properly, so I'd be up for shoving it into nix-daemon.

Ericson2314 commented 1 year ago

Triaged in the Nix team meeting:

@thufschmitt: the only thing we could do is building the installer in a way that it doesn't touch files that will be messed up by macOS updates
- there is no obvious way how to do it
to discuss, maybe we can solicit help from people close to macOS and Apple

nixos-discourse commented 1 year ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-03-03-nix-team-meeting-minutes-37/25998/1

ajmas commented 1 year ago

Would using a LaunchDaemon be an alternative solution to modifying the fstab directly (for the fstab portion of the issue). Have looked at this as a possibility in in issue #7106 , though it is still a bit of an experiment.

abathur commented 1 year ago

For later travelers, I'm fairly certain that the macOS update did not remove your fstab. The fstab ~issue noted by a couple of late comments in this thread is a side effect of trying to fix the installation by re-running the installer (which is not currently idempotent) without completely uninstalling first.

If you run into the core problem here (macOS update overwriting the shell init hook for ZSH) and tried running the installer without uninstalling, you'll need follow the uninstall instructions before reinstalling it: https://nixos.org/manual/nix/stable/installation/installing-binary.html#macos

Edit: someone in matrix confirmed that they installed 13.3 on march 27 2023 and it did not remove fstab.

abathur commented 1 year ago

I'm responding here to part of a comment from another issue because it's more on-topic here. The relevant bit:

Edit: just looking at the zsh files and would one of the files such as /etc/zprofile by a suitable alternative, though again would need to check what files Apple indicates as being safe from overwrites?

I'm not really sure. If there's a problem with it, it'll probably be because zshrc files should get sourced for any interactive shell, while zprofile will only get sourced for login shells. If it is viable, I agree that we should get confirmation that updates don't clobber it before we try to change the installer.

It's a bit of an aside, but there has also been some mumbling (throughout this thread, and in the installer workgroup meetings) about Nix-side alternatives that roughly boil down to:

making the nix CLI/daemon itself able to detect and at least warn about if not cure problems like this
working toward reducing the scope of these init files at the Nix level (by volume they mostly deal with setting up cert envs) since this may help fix some other issues and might open up some other other paths to fixing this one (though I suspect the way Apple uses path_helper in /etc/zprofile will keep that from yielding any better options)

ajmas commented 1 year ago

BTW I decided to ask the question about zsh related files on StackExchange, and it looks like other people deal with the same headache, as part of their sysadmin work:

https://apple.stackexchange.com/questions/458026/which-etc-zsh-related-files-are-safe-from-os-update-overwrites

jamesreprise commented 1 year ago

I experienced this upgrading from Ventura 13.3 to the beta Ventura 13.4. I can confirm the fix as mentioned above by many worked for me - appending the following to '/etc/zshrc':

# Nix
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
  source '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End Nix

I had aliased so many commands to programs installed through Nix that zsh was unusable, I had to use bash to actually edit /etc/zshrc.

iFreilicht commented 1 year ago

@ajmas from a comment to the answer:

launchd task / cron task that checks for the required edits, exits quietly or removes itself if there's no work to do.

Something like this seems to be the only somewhat reliable option. Maybe nix-daemon itself could do that?

pmarreck commented 1 year ago

I see that the nix-daemon.sh script prepends PATH items without checking to see if they're in PATH already. I know this is pretty common practice but the idea of duplicating PATH entries ad nauseam always bothered me so I wrote this Bash function to prepend but first eliminate the relevant entry from PATH if it's already there, making this operation idempotent. The function also works with other PATH-like variables such as LD_LIBRARY_PATH etc.

# function to prepend paths in an idempotent way
prepend_path() {
  local dir="${1%/}"     # discard trailing slash
  local var="${2:-PATH}"
  if [ -z "$dir" ]; then
    echo "Usage: prepend_path <path_to_prepend> [name_of_path_var]" >&2
    return 2 # incorrect usage return code, may be an informal standard
  fi
  local newvalue=${!var}
  [[ $newvalue =~ ^$dir: ]] && return # quit if value already starts with $dir
  newvalue=${newvalue%:$dir}          # remove $dir from end of path
  newvalue=${newvalue//:$dir:/:}      # remove $dir from middle of path
  # prepend the new entry
  export ${var}="$dir:$newvalue"
}

alper commented 1 year ago

Hm there must be a way to have the shell hook somewhere that does not get overwritten, brew e.g. puts it in the users shell env by printing a snipped people have to add themselfs.

I run fish and well… colour me surprised that the Nix installation had messed with /etc/zshrc and that a change in that file has an impact on my shell which is not zsh.

I think asking people to add it themselves to the shell they use is the way to go.

NixOS / nix

macOS updates often break nix installation (updates replace path-hooks on multi-user install) #3616

4169 seems to have a more robust discussion on ~zsh-specific fixes, and I'm trying to fish around for someone to add it to the 2.4 milestone; if I find someone, that might be a good place to pull on the thread...

4169 got closed without dealing with this part of the problem.