chore(nix): cleanup

[Jint-lzxy]

This commit introduced three changes:

1. Opt for a shallow clone whenever we run the update_flake workflow since there's no need to retrive past commits.
2. Deleted result from .gitignore bc I couldn't track down its source (correct me if I'm wrong!)
3. Ensure either XCode or the CLT is installed on macOS before running otool. These aren't included by default on a fresh macOS install, and using xcrun can sort out those dependency snags.

btw @CharlesChiuGit @misumisumi do u think it's worth considering enforcing library policies for the pre-built binaries that Mason uses on macOS, similar to our approach on NixOS? This enforcement could involve setting DYLD_*_LIBRARY_PATH or utilizing install_name_tool (see manual below). One potential concern I see with this change is that macOS often includes software that hasn't been updated in a long time, so transitioning to a new version could potentially cause disruptions.

man install_name_tool

```console INSTALL_NAME_TOOL(1) General Commands Manual INSTALL_NAME_TOOL(1) NAME install_name_tool - change dynamic shared library install names SYNOPSIS install_name_tool [-change old new ] ... [-rpath old new ] ... [-add_rpath new ] ... [-delete_rpath new ] ... [-id name] file DESCRIPTION Install_name_tool changes the dynamic shared library install names and or adds, changes or deletes the rpaths recorded in a Mach-O binary. For this tool to work when the install names or rpaths are larger the binary should be built with the ld(1) -headerpad_max_install_names option. -change old new Changes the dependent shared library install name old to new in the specified Mach-O binary. More than one of these options can be specified. If the Mach-O binary does not contain the old install name in a specified -change option the option is ignored. -id name Changes the shared library identification name of a dynamic shared library to name. If the Mach-O binary is not a dynamic shared library and the -id option is specified it is ignored. -rpath old new Changes the rpath path name old to new in the specified Mach-O binary. More than one of these options can be specified. If the Mach-O binary does not contain the old rpath path name in a specified -rpath it is an error. -add_rpath new Adds the rpath path name new in the specified Mach-O binary. More than one of these options can be specified. If the Mach-O binary already contains the new rpath path name specified in -add_rpath it is an error. -delete_rpath old deletes the rpath path name old in the specified Mach-O binary. More than one of these options can be specified. If the Mach-O binary does not contains the old rpath path name specified in -delete_rpath it is an error. SEE ALSO dyld(1), ld(1), otool(1) Apple, Inc. March 4, 2009 INSTALL_NAME_TOOL(1) ```

man dyld

```console DYLD(1) General Commands Manual DYLD(1) NAME dyld - the dynamic linker SYNOPSIS DYLD_FRAMEWORK_PATH DYLD_FALLBACK_FRAMEWORK_PATH DYLD_VERSIONED_FRAMEWORK_PATH DYLD_LIBRARY_PATH DYLD_FALLBACK_LIBRARY_PATH DYLD_VERSIONED_LIBRARY_PATH DYLD_IMAGE_SUFFIX DYLD_INSERT_LIBRARIES DYLD_PRINT_TO_FILE DYLD_PRINT_LIBRARIES DYLD_PRINT_LOADERS DYLD_PRINT_SEARCHING DYLD_PRINT_APIS DYLD_PRINT_BINDINGS DYLD_PRINT_INITIALIZERS DYLD_PRINT_SEGMENTS DYLD_PRINT_ENV DYLD_SHARED_REGION DYLD_SHARED_CACHE_DIR DESCRIPTION The dynamic linker (dyld) checks the following environment variables during the launch of each process. Note: If System Integrity Protection is enabled, these environment variables are ignored when executing binaries protected by System Integrity Protection. DYLD_FRAMEWORK_PATH This is a colon separated list of directories that contain frameworks. The dynamic linker searches these directories before it searches for the framework by its install name. It allows you to test new versions of existing frameworks. (A framework is a library install name that ends in the form XXX.framework/Versions/A/XXX or XXX.framework/XXX, where XXX and A are any name.) For each framework that a program uses, the dynamic linker looks for the framework in each directory in DYLD_FRAMEWORK_PATH in turn. If it looks in all those directories and can't find the framework, it uses whatever it would have loaded if DYLD_FRAMEWORK_PATH had not been set. Use the -L option to otool(1) to discover the frameworks and shared libraries that the executable is linked against. DYLD_FALLBACK_FRAMEWORK_PATH This is a colon separated list of directories that contain frameworks. If a framework is not found at its install path, dyld uses this as a list of directories to search for the framework. For new binaries (Fall 2023 or later) there is no default fallback. For older binaries, their is a default fallback search path of: /Library/Frameworks:/System/Library/Frameworks DYLD_VERSIONED_FRAMEWORK_PATH This is a colon separated list of directories that contain potential override frameworks. The dynamic linker searches these directories for frameworks. For each framework found dyld looks at its LC_ID_DYLIB and gets the current_version and install name. Dyld then looks for the framework at the install name path. Whichever has the larger current_version value will be used in the process whenever a framework with that install name is required. This is similar to DYLD_FRAMEWORK_PATH except instead of always overriding, it only overrides if the supplied framework is newer. Note: dyld does not check the framework's Info.plist to find its version. Dyld only checks the -current_version number supplied when the framework was created. DYLD_LIBRARY_PATH This is a colon separated list of directories that contain libraries. The dynamic linker searches these directories before it searches the default locations for libraries. It allows you to test new versions of existing libraries. For each dylib that a program uses, the dynamic linker looks for its leaf name in each directory in DYLD_LIBRARY_PATH. Use the -L option to otool(1) to discover the frameworks and shared libraries that the executable is linked against. DYLD_FALLBACK_LIBRARY_PATH This is a colon separated list of directories that contain libraries. If a dylib is not found at its install path, dyld uses this as a list of directories to search for the dylib. For new binaries (Fall 2023 or later) there is no default. For older binaries, their is a default fallback search path of: /usr/local/lib:/usr/lib. DYLD_VERSIONED_LIBRARY_PATH This is a colon separated list of directories that contain potential override libraries. The dynamic linker searches these directories for dynamic libraries. For each library found dyld looks at its LC_ID_DYLIB and gets the current_version and install name. Dyld then looks for the library at the install name path. Whichever has the larger current_version value will be used in the process whenever a dylib with that install name is required. This is similar to DYLD_LIBRARY_PATH except instead of always overriding, it only overrides is the supplied library is newer. DYLD_IMAGE_SUFFIX This is set to a string of a suffix to try to be used for all shared libraries used by the program. For libraries ending in ".dylib" the suffix is applied just before the ".dylib". For all other libraries the suffix is appended to the library name. This is useful for using conventional "_profile" and "_debug" libraries and frameworks. DYLD_INSERT_LIBRARIES This is a colon separated list of additional dynamic libraries to load before the ones specified in the program. If instead, your goal is to substitute a library that would normally be loaded, use DYLD_LIBRARY_PATH or DYLD_FRAMEWORK_PATH instead. DYLD_PRINT_TO_FILE This is a path to a (writable) file. Normally, the dynamic linker writes all logging output (triggered by DYLD_PRINT_* settings) to file descriptor 2 (which is usually stderr). But this setting causes the dynamic linker to write logging output to the specified file. DYLD_PRINT_ENV If set, causes dyld to print a line of key=valule for each enviroment variable in the process. DYLD_PRINT_LIBRARIES If set, causes dyld to print a line for each mach-o image loaded into a process. This is useful to make sure that the use of DYLD_LIBRARY_PATH is getting what you want. DYLD_PRINT_LOADERS If set, causes dyld to print a line whether each image is tracked by a JustInTimeLoader or a PrebuiltLoader. Additionally, it prints if a PrebuiltLoaderSet was used to launch the process or if a PrebuiltLoader was written to make the next launch faster. DYLD_PRINT_SEARCHING If set, causes dyld to print a line about each file system path checked when searching for an image to load. DYLD_PRINT_INITIALIZERS If set, causes dyld to print out a line when running each initializer in every image. Initializers run by dyld include constructors for C++ statically allocated objects, functions marked with __attribute__((constructor)), and -init functions. DYLD_PRINT_APIS If set, causes dyld to print a line whenever a dyld API is called (e.g. dlopen()). DYLD_PRINT_SEGMENTS If set, causes dyld to print out a line containing the name and address range of each mach- o segment that dyld maps. In addition it prints information about if the image was from the dyld shared cache. DYLD_PRINT_BINDINGS If set, causes dyld to print a line each time a symbolic name is bound. DYLD_SHARED_REGION This can be "use" (the default) or "private". Setting it to "private" tells dyld to remove the shared region from the process address space and mmap() back in a private copy of the dyld shared cache in the shared region address range. This is only useful if the shared cache on disk has been updated and is different than the shared cache in use. DYLD_SHARED_CACHE_DIR This is a directory containing dyld shared cache files. This variable can be used in conjunction with DYLD_SHARED_REGION=private to run a process with an alternate shared cache. DYNAMIC LIBRARY LOADING Unlike many other operating systems, Darwin does not locate dependent dynamic libraries via their leaf file name. Instead the full path to each dylib is used (e.g. /usr/lib/libSystem.B.dylib). But there are times when a full path is not appropriate; for instance, may want your binaries to be installable in anywhere on the disk. To support that, there are three @xxx/ variables that can be used as a path prefix. At runtime dyld substitutes a dynamically generated path for the @xxx/ prefix. @executable_path/ This variable is replaced with the path to the directory containing the main executable for the process. This is useful for loading dylibs/frameworks embedded in a .app directory. If the main executable file is at /some/path/My.app/Contents/MacOS/My and a framework dylib file is at /some/path/My.app/Contents/Frameworks/Foo.framework/Versions/A/Foo, then the framework load path could be encoded as @executable_path/../Frameworks/Foo.framework/Versions/A/Foo and the .app directory could be moved around in the file system and dyld will still be able to load the embedded framework. @loader_path/ This variable is replaced with the path to the directory containing the mach-o binary which contains the load command using @loader_path. Thus, in every binary, @loader_path resolves to a different path, whereas @executable_path always resolves to the same path. @loader_path is useful as the load path for a framework/dylib embedded in a plug-in, if the final file system location of the plugin-in unknown (so absolute paths cannot be used) or if the plug-in is used by multiple applications (so @executable_path cannot be used). If the plug-in mach-o file is at /some/path/Myfilter.plugin/Contents/MacOS/Myfilter and a framework dylib file is at /some/path/Myfilter.plugin/Contents/Frameworks/Foo.framework/Versions/A/Foo, then the framework load path could be encoded as @loader_path/../Frameworks/Foo.framework/Versions/A/Foo and the Myfilter.plugin directory could be moved around in the file system and dyld will still be able to load the embedded framework. @rpath/ Dyld maintains a current stack of paths called the run path list. When @rpath is encountered it is substituted with each path in the run path list until a loadable dylib if found. The run path stack is built from the LC_RPATH load commands in the depencency chain that lead to the current dylib load. You can add an LC_RPATH load command to an image with the -rpath option to ld(1). You can even add a LC_RPATH load command path that starts with @loader_path/, and it will push a path on the run path stack that relative to the image containing the LC_RPATH. The use of @rpath is most useful when you have a complex directory structure of programs and dylibs which can be installed anywhere, but keep their relative positions. This scenario could be implemented using @loader_path, but every client of a dylib could need a different load path because its relative position in the file system is different. The use of @rpath introduces a level of indirection that simplifies things. You pick a location in your directory structure as an anchor point. Each dylib then gets an install path that starts with @rpath and is the path to the dylib relative to the anchor point. Each main executable is linked with -rpath @loader_path/zzz, where zzz is the path from the executable to the anchor point. At runtime dyld sets it run path to be the anchor point, then each dylib is found relative to the anchor point. SEE ALSO dyldinfo(1), ld(1), otool(1) Apple Inc. June 1, 2020 DYLD(1) ```

[misumisumi]

Deleted result from .gitignore bc I couldn't track down its source (correct me if I'm wrong!)

This was added to prevent accidentally committing this because when you run nix build without specifying --out-link, an artifact is generated in ./result. You can delete this if you don't need it.

[misumisumi]

btw @CharlesChiuGit @misumisumi do u think it's worth considering enforcing library policies for the pre-built binaries that Mason uses on macOS, similar to our approach on NixOS?

Hmm. I can't answer that question because I don't have a Mac environment at hand. If it's a general question that doesn't depend on nixpkgs (such as when you installed dependencies with homebrew), I think it's a problem with mason.nvim or a symmetric package.

---

We can check if nixpkgs is linked with check-linker. check-linker checks whether dependencies are linked to /nix/store in Nix environments, regardless of whether they are NixOS or non-NixOS. Although it was not explained in #1250 , this script will scan $XDG_DATA_HOME/$NVIM_APPNAME/mason/bin under NVIM_APPNAME if it is specified, and $XDG_DATA_HOME/nvim/mason/bin if not.

[misumisumi]

This enforcement could involve setting DYLD_*_LIBRARY_PATH or utilizing install_name_tool (see manual below).

If we need DYLD_*_LIBRARY_PATH settings, it's easy to add them too. We only add environment valiable or extraWrapperArgs in nixos/neovim/default.nix.

[CharlesChiuGit]

tbh, i just knew we can set DYLD_*_LIBRARY_PATH or utilizing install_name_tool, newbie me lol. I think specifying DYLD_*_LIBRARY_PATH sounds like a gd idea to me, so we can make things more standardized.

One potential concern I see with this change is that macOS often includes software that hasn't been updated in a long time

i think this should be fine since those fundamental libs generally have gd backward compatibility?( correct me if i'm wrong lol

[Cyberczy]

@misumisumi Can u tell me how to create a package using Nix, or what should I learn?

[misumisumi]

You can learn from official nixpkg manual. There are also articles written by me, although they are in Japanese. You can learn further understanding and application by referring to the actual package of NixOS/nixpkgs.

[Jint-lzxy]

@misumisumi @CharlesChiuGit Apologies for the delayed response and the oversight in explaining the issue! Let me provide further detail on how shared libraries are located on macOS (Darwin). Unlike many other Unix-like operating systems, Darwin's dynamic linker, dyld, does not find dependent shared libraries based on their leaf filename (i.e., the one obtained using basename). Instead, it directly utilizes the paths hardcoded in the executable. For instance, consider these two examples (ld and dyld, with simplified output):

$ ldd a.out
    libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000fffff76c0000)

and

$ otool -L a.out
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

The crucial difference is that on Darwin, dyld only[^1] tries to find libSystem.B.dylib at /usr/lib/libSystem.B.dylib. If it cannot find libSystem under /usr/lib/, it directly fails, displaying an error message like this:

dyld[48101]: Library not loaded: /usr/lib/libSystem.B.dylib
  Referenced from: <08197395-BB6E-33C2-BB73-C47F2C1C672E> /private/var/folders/jy/91xg_8j9169dydct2k4r7yw80000gn/T/a.out.cdBaJwRHju/a.out
  Reason: tried: '/usr/lib/libSystem.B.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/libSystem.B.dylib' (no such file), '/usr/lib/libSystem.B.dylib' (no such file, not in dyld cache)
[1]    48101 abort      ./a.out

Which is why I mentioned earlier in https://github.com/ayamir/nvimdots/pull/1252#issue-2281283101 that on Darwin, we may only[^2] either use install_name_tool to change the hardcoded path on Mason's pre-built executables[^3] to point to the Nix store, or specify the DYLD_*_LIBRARY_PATH environment variable to make dyld operate akin to ld, dynamically locating libraries during runtime with specified paths. Hopefully this clears up some confusion regarding shared libraries on macOS :D

[^1]: Actually dyld would also search for libSystem.B.dylib inside its shared cache using symbol stubs, and scrutinize several MachO-specific load commands like LC_RPATH in an attempt to find it, but these complexities are all unnecessary here. [^2]: Package managers following the "traditional" directory hierarchy are usually considered, with their paths automatically included in the executables' @rpath. These paths are prioritized first where applicable, such as /usr/local/lib/ for Homebrew formulas. [^3]: Since they're not utilizing our dependencies installed with Nix, imo it diminishes the value of using Nix on macOS.

[Jint-lzxy]

i think this should be fine since those fundamental libs generally have gd backward compatibility?( correct me if i'm wrong lol

@CharlesChiuGit Yeah u're absolutely right lol. Actually, what I meant to say was that those "new" features that come with some packages could alter the way Neovim operates. For instance, ncurses 6 introduces a bunch of new features (and databases) that ncurses 5 doesn't have. So, if we now instruct some executable to use ncurses 6 (macOS ships with a really ancient version of ncurses 5), the display could look different (like with colors).

[CharlesChiuGit]

so set DYLD_*_LIBRARY_PATH to nix's lib should make things more compatible w/ new features? for example, set DYLD_*_LIBRARY_PATH to .local/state/nix/profile/lib(xdg = true) ,.nixprofile/lib(xdg = false) or sth similar like https://github.com/ayamir/nvimdots/blob/e3e5797d1694026ed6babfdad6cc89b22c5cbe2c/nixos/neovim/default.nix#L106-L111?

[Jint-lzxy]

so set DYLD_*_LIBRARY_PATH to nix's lib should make things more compatible w/ new features?

Ye and IMO this should be the most straightforward method. However, the drawback here is that other processes spawned by Neovim (e.g., lazygit) that aren't managed by Nix could also be impacted during library resolution. On the other hand, using install_name_tool is more intricate but carries fewer side effects.

[misumisumi]

This discussion is similar to dealing with binary packages in NixOS. When using binary on NixOS, there is a way to rewrite the path to dylib using patchelf. (This is other method adopted here., more details Packaging/Binaries) I previously used a method to automatically rewrite dylib using mason.nvim's hook. I settled on the current method because it was incomplete in various ways, but it should be a start for discussion it. More details, under article (This is written by Japanese.) NixOSでmason.nvimで入れたLSP serverを使えるようにする

[misumisumi]

However, the drawback here is that other processes spawned by Neovim (e.g., lazygit) that aren't managed by Nix could also be impacted during library resolution.

That's true. In addition, quickfix and command execution in the internal terminal are affected. However, I decided that this problem could be avoided by using a multiplexer, so I adopted the current method.

[misumisumi]

On the other hand, using install_name_tool is more intricate but carries fewer side effects.

IMO, if we use this method to solve the problem on macOS, we should be able to use almost the same method on NixOS, so it is better to replace both.

[CharlesChiuGit]

just a side note, i just found out that upgrading macOS will break nix, since it regenerate /etc/zshrc, but it's also easy to fix. see the fix here.

maybe jint will experiences similar issues lol

[Jint-lzxy]

just a side note, i just found out that upgrading macOS will break nix, since it regenerate /etc/zshrc, but it's also easy to fix.

@CharlesChiuGit I just spotted this in my inbox and wanted to quickly mention that I managed to "circumvent" the issue by setting the system immutable flag on /etc/zshrc:

sudo chflags simmutable /etc/zshrc

bc I discovered that the file's content hasn't changed since Catalina (the first version to use zsh as the default shell). AFAIU this tweak ensures the installer tool (the underlying exec for system software updates) can't undo my changes with each update. But to make any modifications, the file has to be temporarily "unlocked" using:

sudo chflags nosimmutable /etc/zshrc

See chflags(1), installer(8).

[CharlesChiuGit]

damn, i completely brake my nix environment and can't get it fixed 🥲 (reinstalled few times lol

ok, i finally got it fixed lol i found this interesting repo that claims to prevent nix to break when doing OS upgrade, testing it now. https://github.com/determinatesystems/nix-installer

[misumisumi]

The section on nixos in the wiki is out of date and needs to be updated Also, lazy.nvim is not designed to be a read-only file, so the current arrangement will update the file, but it will give an error. Should we fix this? I will be busy until mid-June, so I may not be able to respond much

[Jint-lzxy]

I'm still new to Nix(OS) so plz correct me if any of the following is conceptually wrong!

I settled on the current method because it was incomplete in various ways, but it should be a start for discussion it. ... IMO, if we use this method to solve the problem on macOS, we should be able to use almost the same method on NixOS, so it is better to replace both.

So iiuc depending on how much extra work is needed, here are our options:

1. Utilize environment variables to help the linker locate the necessary dependencies. * The drawback here is that it impacts every subprocess launched by Neovim, and there's no simple workaround. However, I'm not entirely convinced this is a critical concern.
2. Skip the entire package system (aka Mason) and utilize the executables provided by the native package manager instead. * The downside is we might have to enhance the current LSP "framework" significantly to handle locally installed executables first (and bypass Mason when necessary). Also, not all Mason packages might be available through native package managers, and some users might have a messy local storage, leading to unexpected issues.
3. Modify the executables directly using the appropriate tools to ensure permanent changes. * The downside is that it could be challenging to reliably parse the dynamic linker's output and then systematically feed it to another program (e.g., nix path-info).

Honestly, I actually lean towards option 2 bc it's the most stable one and also tries to be as Nix-ish as possible. It can accommodate existing executables and also conserve disk space. What do u guys think?

The section on nixos in the wiki is out of date and needs to be updated

I suggest postponing this until we reach a consensus on the implementation. IMO it's probable that our implementation will change in the future, and documenting those volatile configs doesn't sit well with anyone at all.

Also, lazy.nvim is not designed to be a read-only file, so the current arrangement will update the file, but it will give an error.

I think if we're dead set on making the entire config immutable, we could have GitHub actions handle the updates. Then, on the end user's local setup, they could simply do a :Lazy restore (as opposed to :Lazy sync/update) to stay in sync with the changes. This approach won't mess with the lockfile and could still guarantee periodic lockfile updates (which we could control using an actions cronjob).

Alternatively, if we're not too strict about restricted write access, we could stash the lockfile in a separate location[^1] like stdpath("state")/lazy-lock.json, keeping it apart from our config. This way, the existing config remains untouched, and all we have to do is tweak the install script to reroute the lockfile in our repository to this designated location for machines adhering to the traditional filesystem hierarchy.

[^1]: Specifically, by setting lockfile = vim.fn.stdpath("state") .. "/lazy-lock.json".

[misumisumi]

Honestly, I actually lean towards option 2 bc it's the most stable one and also tries to be as Nix-ish as possible.

I also think this is the best. However, as you say, major changes are required around the LSP configuration. This is because mason-lsp-config cannot be used.

---

IMO, is better creating plugin for bypass when exsting global pkg. Also valid for apt, pacman, homebrew (mac), scoop (win). If it has already been installed globally using the mason.nvim API, it would be nice if it could be bypassed as installed. Lack of experience and time to create plugins. Other option, configuration native lsp. (without mason ecosystem.)

[misumisumi]

they could simply do a :Lazy restore

I recently learned this. Maybe you don't need to worry too much about the lazy.nvim error.

[CharlesChiuGit]

https://github.com/nix-community/neovim-nightly-overlay/issues/533

i wonder if this affect us.

[misumisumi]

I think there were no problems as of last Friday, but I'll check.

[Jint-lzxy]

I also think this is the best. However, as you say, major changes are required around the LSP configuration. This is because mason-lsp-config cannot be used.

I've got some free time now and can start working on this. But I probably won't finish before https://github.com/ayamir/nvimdots/pull/1260 gets merged in tho.

IMO, is better creating plugin for bypass when exsting global pkg. Also valid for apt, pacman, homebrew (mac), scoop (win). If it has already been installed globally using the mason.nvim API, it would be nice if it could be bypassed as installed. Lack of experience and time to create plugins. Other option, configuration native lsp. (without mason ecosystem.)

Thanks for this brilliant idea! It didn't even cross my mind lol

[Jint-lzxy]

Given that #1293 is now open, shall we merge this PR first and continue our discussion there?