Managing files without symlinks

[mightybyte]

There are some configuration files that I would like to somehow manage with home-manager, but that also get changed by other tools or that you don't want to put in your home-manager config. An example of the former is the .spacemacs file which most people update directly from spacemacs. An example of the latter is the .ssh/config file which I often want to augment with servers that I do not want to put into my home-manager config (because I would like to be able to make my home-manager config public). The problem is that home-manager creates symlinks to read-only files in the nix store which causes problems for these two use cases. Is there a way this issue could be worked around?

[infinisil]

The only way I have found to allow that is to load a mutable config from the immutable on. This requires the config file language to have include-like functionality. As an example, I'm doing this for my emacs config:

{
  home.files.".emacs.d/init.el".text = ''
    (message "Hi!")

    (setq custom-file "${toString ./custom.el}")
    (load custom-file)
  '';
}

The toString will make it so that it doesn't import custom.el into an immutable /nix/store path, but rather expands to /path/to/config/custom.el. This allows you to change the custom.el file via emacs 'customize' interface, while still keeping the other part immutable.

This can also be used to have a tmp.el file you include for temporary quick changes, which you can then transfer to home-manager to make them 'permanent'.

[rycee]

One possibility might be to use an unsupported trick: to set the file source to a string containing the absolute path to the file that should be maintained outside the Nix store. For example,

home.file.".spacemacs".source = "${config.home.homeDirectory}/where/i/keep/my/spacemacs";

Running home-manager switch should produce a symlink ~/.spacemacs pointing to a symlink within the Home Manager generation, which in turn will point to ~/where/i/keep/my/spacemacs which is entirely your responsibility to manage.

I'm not sure I would recommend using this method, instead perhaps simply add an activation script that creates the symlink yourself? It would make it more explicit:

home.activation.linkMyStuff = dag.entryAfter [ "writeBoundary" ] ''
  ln -sf $HOME/where/i/keep/my/spacemacs $HOME.spacemacs
'';

[teto]

Being more versatile on the generated paths should be mandatory for better nixos/home-manager integration (a module should be able to be installed systemwide e.g., /etc/vimrc vs userwide ~/.vimrc). Also depending on the programs I would rather include a fixed config or have a config include a generated config (in case you sync your dotfiles with non-nix distros).

[ivanbrennan]

Also depending on the programs I would rather include a fixed config or have a config include a generated config (in case you sync your dotfiles with non-nix distros).

I agree with this point especially.

[piegamesde]

because I would like to be able to make my home-manager config public

A trick how to do this is to create some secrets folder, add it to the gitignore and put all the non-shared configuration into it. The files in there should be sourced in a way that they act like an overlay, the config should still be able to build without them.

I've not done this myself yet, but I know a few public NixOS configs that work this way.

[stale[bot]]

Thank you for your contribution! I marked this issue as stale due to inactivity. If this remains inactive for another 7 days, I will close this issue. Please read the relevant sections below before commenting.

If you are the original author of the issue

* If this is resolved, please consider closing it so that the maintainers know not to focus on this. * If this might still be an issue, but you are not interested in promoting its resolution, please consider closing it while encouraging others to take over and reopen an issue if they care enough. * If you know how to solve the issue, please consider submitting a Pull Request that addresses this issue.

If you are not the original author of the issue

* If you are also experiencing this issue, please add details of your situation to help with the debugging process. * If you know how to solve the issue, please consider submitting a Pull Request that addresses this issue.

Memorandum on closing issues

If you have nothing of substance to add, please refrain from commenting and allow the bot close the issue. Also, don't be afraid to manually close an issue, even if it holds valuable information.

Closed issues stay in the system for people to search, read, cross-reference, or even reopen--nothing is lost! Closing obsolete issues is an important way to help maintainers focus their time and effort.

[solson]

I just started using home-manager, and one thing I really hope to achieve is to symlink some mutable configuration files from my home-manager/dotfiles repo into my home directory using the exact same logic home-manager uses to avoid overwriting data accidentally. It's easy enough to put ln -s in an activation script, but I would love to hook into the safe, managed symlinking system that home-manager already implements.

[rycee]

The comment i made in https://github.com/nix-community/home-manager/issues/257#issuecomment-388146775 is outdated. The way to do it now is to use an "out of store symlink". Something like

home.file."file.foo".source = config.lib.file.mkOutOfStoreSymlink ./path/to/file/to/link;

should produce a symlink at ~/file.foo that will lead to the indicated file's original location, i.e., not into the Nix store.

If you remove the line from your configuration then the next switch should remove ~/file.foo.

[teto]

I've added it to the FAQ https://github.com/nix-community/home-manager/wiki/FAQ maybe we can close ?

[solson]

I didn't realize mkOutOfStoreSymlink already existed. That works perfectly, thanks!

I'm okay with closing at least as far as my above usecase is concerned.

[stale[bot]]

Thank you for your contribution! I marked this issue as stale due to inactivity. If this remains inactive for another 7 days, I will close this issue. Please read the relevant sections below before commenting.

If you are the original author of the issue

* If this is resolved, please consider closing it so that the maintainers know not to focus on this. * If this might still be an issue, but you are not interested in promoting its resolution, please consider closing it while encouraging others to take over and reopen an issue if they care enough. * If you know how to solve the issue, please consider submitting a Pull Request that addresses this issue.

If you are not the original author of the issue

* If you are also experiencing this issue, please add details of your situation to help with the debugging process. * If you know how to solve the issue, please consider submitting a Pull Request that addresses this issue.

Memorandum on closing issues

If you have nothing of substance to add, please refrain from commenting and allow the bot close the issue. Also, don't be afraid to manually close an issue, even if it holds valuable information.

Closed issues stay in the system for people to search, read, cross-reference, or even reopen--nothing is lost! Closing obsolete issues is an important way to help maintainers focus their time and effort.

[oati]

Can we please disable stalebot's auto-close? This makes it impossible to differentiate between unresolved and resolved issues.

[oati]

Or would you prefer that I open duplicate issues, burying all previous discussion on the same problem?

[oati]

It would be great if there was some way to set any home-manager-generated config to be "overridable" so that I could test trivial changes without being forced to rebuild.

The output config would have write permissions, but running the home-manager service would "reset" them to the home-manager-generated config.

This could also allow declarative management of programs that require their configs to be mutable, or programs that "prefer" it (e.g. programs with graphical configuration interfaces).

[oati]

One possible way of implementing this would be to add an option to xdg.configFile and xdg.dataFile that copies files instead of symlinking from the nix store.

[musjj]

Weird, config.lib.file.mkOutOfStoreSymlink ./path/to/file doesn't work for me. It still links to the store.

[ncfavier]

That is expected if you use flakes, which live in the Nix store. You have to specify the absolute path on your file system, e.g. mkOutOfStoreSymlink "/home/you/config/path/to/file".

[musjj]

Thanks, that works for me.

[solson]

I have a fairly robust solution I've been using for that issue:

    runtimeRoot = "/path/to/my/repository";
    runtimePath = path:
      let
        # This is the `self` that gets passed to a flake `outputs`.
        rootStr = toString self;
        pathStr = toString path;
      in
      assert lib.assertMsg
        (lib.hasPrefix rootStr pathStr)
        "${pathStr} does not start with ${rootStr}";
      runtimeRoot + lib.removePrefix rootStr pathStr;

Used like:

        source = mkOutOfStoreSymlink (runtimePath path);

In essence you could think of runtimeRoot as like an extra input to the flake, since the source location itself isn't allowed to leak into flake evaluation. With an idea like https://github.com/NixOS/nix/issues/5663, it could probably become a literal flake input in the future.

[NovaViper]

I have a fairly robust solution I've been using for that issue:

    runtimeRoot = "/path/to/my/repository";
    runtimePath = path:
      let
        # This is the `self` that gets passed to a flake `outputs`.
        rootStr = toString self;
        pathStr = toString path;
      in
      assert lib.assertMsg
        (lib.hasPrefix rootStr pathStr)
        "${pathStr} does not start with ${rootStr}";
      runtimeRoot + lib.removePrefix rootStr pathStr;

Used like:

        source = mkOutOfStoreSymlink (runtimePath path);

In essence you could think of runtimeRoot as like an extra input to the flake, since the source location itself isn't allowed to leak into flake evaluation. With an idea like NixOS/nix#5663, it could probably become a literal flake input in the future.

Hey where exactly are you declaring this? I'm dealing with the same issue and want to declare this within my flake setup for my dotfiles too

[solson]

@NovaViper runtimeRoot and runtimePath are defined (along with other helpers) in an attr at the top level of my flake outputs (let's call it dotfilesLib), and then I pass that into each Home Manager and NixOS config like this:

  homeConf = host: (home-manager.lib.homeManagerConfiguration {
    inherit pkgs;
    extraSpecialArgs = { inherit dotfilesLib; };
    modules = [ ./machines/${host}/home ];
  });

Then, the ./machines/${host}/home/default.nix file (or any other module it imports) can grab it from the args on the first line of the file:

{ dotfilesLib, config, lib, pkgs, ... }:

And call it like:

  source = mkOutOfStoreSymlink (dotfilesLib.runtimePath path);

[NovaViper]

@NovaViper runtimeRoot and runtimePath are defined (along with other helpers) in an attr at the top level of my flake outputs (let's call it dotfilesLib), and then I pass that into each Home Manager and NixOS config like this:

  homeConf = host: (home-manager.lib.homeManagerConfiguration {
    inherit pkgs;
    extraSpecialArgs = { inherit dotfilesLib; };
    modules = [ ./machines/${host}/home ];
  });

Then, the ./machines/${host}/home/default.nix file (or any other module it imports) can grab it from the args on the first line of the file:

{ dotfilesLib, config, lib, pkgs, ... }:

And call it like:

  source = mkOutOfStoreSymlink (dotfilesLib.runtimePath path);

Hey thank you for clarifying! I started adding it into my flake but I'm still a little confused with how to declare it in the flake.nix file, I wrote it out like below

  outputs = { self, nixpkgs, home-manager, ... }@inputs:
    let
      inherit (self) outputs;
      lib = nixpkgs.lib // home-manager.lib;
      systems = [ "x86_64-linux" ];
      forEachSystem = f: lib.genAttrs systems (sys: f pkgsFor.${sys});
      pkgsFor = nixpkgs.legacyPackages; 
    in {
      inherit lib;

      dotfilesLib = {
        runtimeRoot = "/path/to/my/repository";
        runtimePath = path:
          let
            # This is the `self` that gets passed to a flake `outputs`.
            rootStr = toString self;
            pathStr = toString path;
          in assert lib.assertMsg (lib.hasPrefix rootStr pathStr)
            "${pathStr} does not start with ${rootStr}";
            runtimeRoot + lib.removePrefix rootStr pathStr; #<<< HERE IS THE PROBLEMATIC SECTION
      };
... Removed the rest of file because unnecessary for this snippet

And where I highlighted the problematic part, it says here that the runtimeRoot variable is undefined, despite it being declared just before the section. I'm unsure of how to fix this issue or if I'm even defining the attributes correctly since I'm fairly new to Nix and still trying to grasp much of the conceptual aspects of NixOS.

[ncfavier]

Either use dotfilesLib = rec { to make the attributes recursive, or use self.dotfilesLib.runtimeRoot.

[solson]

In my case, dotfilesLib is defined in the let ... part, not the in { ... } part, but a number of different variations could work.

[NovaViper]

@solson @ncfavier Thank you both! I made the changes and added it into one of my modules as a test, however I'm noticing it's not sourcing the folder correctly, making a blank source. Initial definition in flake.nix

  outputs = { self, nixpkgs, home-manager, ... }@inputs:
    let
      inherit (self) outputs;
      lib = nixpkgs.lib // home-manager.lib;
      systems = [ "x86_64-linux" ];
      forEachSystem = f: lib.genAttrs systems (sys: f pkgsFor.${sys});
      pkgsFor = nixpkgs.legacyPackages;
      dotfilesLib = rec {
        runtimeRoot = "/home/novaviper/Desktop";
        runtimePath = path:
          let
            # This is the `self` that gets passed to a flake `outputs`.
            rootStr = toString self;
            pathStr = toString path;
          in assert lib.assertMsg (lib.hasPrefix rootStr pathStr)
            "${pathStr} does not start with ${rootStr}";
          runtimeRoot + lib.removePrefix rootStr pathStr;
      };
    in {
.... more not included

Definition within the modules

{ config, lib, pkgs, dotfilesLib, ... }:

{
  xdg.configFile = {
    "PrusaSlicer/printer" = {
      #source = config.lib.file.mkOutOfStoreSymlink ../../../dots/doom;
      source = config.lib.file.mkOutOfStoreSymlink
        (dotfilesLib.runtimePath home/novaviper/dots/PrusaSlicer/printer);
      recursive = true;
    };
... Rest not included

The repo's structure (removed a good bit of the folders listed to make the block not nearly as long)

Desktop/nix-config (root folder)
├── flake.lock
├── flake.nix
├── home
│  └── novaviper
│     ├── dots
│     │  ├── doom
│     │  │  ├── config.org
│     │  │  ├── fonts.el
│     │  ├── PrusaSlicer
│     │  │  ├── filament
│     │  │  │  ├── Generic FLEX - Copy.ini
│     │  │  │  ├── Generic FLEX @Creality.ini
│     │  │  ├── physical_printer
│     │  │  │  └── Athena.ini
│     │  │  ├── print
│     │  │  │  ├── 0.08 mm SUPERDETAIL (0.4 mm nozzle) @CREALITY - Copy.ini
│     │  │  ├── printer
│     │  │  │  └── Creality Ender-3 Klipper V6 (0.4 mm nozzle).ini
│     │  │  ├── PrusaSlicer.ini
[...]

[ncfavier]

It looks like your runtimeRoot should be /home/novaviper/Desktop/nix-config.

[NovaViper]

I tried that too but it still resulted in a blank symlink

[ncfavier]

What does namei "$XDG_CONFIG_HOME/PrusaSlicer/printer" say?

[NovaViper]

Here (my original comment had ran the command in the wrong directory)

❯ namei "$XDG_CONFIG_HOME/PrusaSlicer/printer"
f: /home/novaviper/.config/PrusaSlicer/printer
 d /
 d home
 d novaviper
 d .config
 d PrusaSlicer
 l printer -> /nix/store/l60fvnkaaidq3bay4lqskdsip80mvgb6-home-manager-files/.config/PrusaSlicer/printer
   d /
   d nix
   d store
   d l60fvnkaaidq3bay4lqskdsip80mvgb6-home-manager-files
   d .config
   d PrusaSlicer
   l printer -> /nix/store/95qxkz6q0p5ay1lmlnb8853bb7vmj2m3-hm_printer
     d /
     d nix
     d store
     l 95qxkz6q0p5ay1lmlnb8853bb7vmj2m3-hm_printer -> /home/novaviper/Desktop/nix-config/home/novaviper/features/productivity/home/novaviper/dots/PrusaSlicer/printer
       d /
       d home
       d novaviper
       d Desktop
       d nix-config
       d home
       d novaviper
       d features
       d productivity
          home - No such file or directory

[solson]

source = config.lib.file.mkOutOfStoreSymlink (dotfilesLib.runtimePath home/novaviper/dots/PrusaSlicer/printer); 

This part looks wrong - the path literal should be relative to the file this code itself is in. e.g. if my /foo/home.nix was using runtimePath for /foo/bar/baz, I would need runtimePath ./bar/baz.

For more context, note that Nix itself expands that path literal to /foo/bar/baz in regular evaluation, but something like /nix/store/...-source/bar/baz in Flake evaluation (if /foo is the flake root), and /nix/store/...-source will be the prefix that runtimePath replaces with runtimeRoot.

[ncfavier]

Right, either use ../../dots/PrusaSlicer/printer, or "home/novaviper/dots/PrusaSilcer/printer" and modify your function so that it expects a runtimeRoot-relative string.

[NovaViper]

Ah, so runtimePath is relative to the module that's evaluating the code that contains it! So essentially when I want to refer to a particular file, I just do the ../.. thing like I've been doing but I also add the runtimePath variable to make to make the flake know where exactly the file is located on the system. Made the changed to the path and got it all working! Thanks for the help @ncfavier @solson !

[NovaViper]

Hey I got one more question again (hopefully this being the last!). I'm trying install Nixos onto my laptop using the config I made but I'm having trouble getting the script to start, mainly because it's running into issues with the dotfilesLib code. I have the flake inside of /mnt (which is where my system is mounted so I can install the os onto it); but the flake says error: /mnt/nix/store/(storeid)-source/home/novaviper/dots/doom does not start with /nix/store/(storeid)-source. I've tried changing the runtimeRoot to /mnt/etc/nixos and all but still can't get it working. @ncfavier @solson

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/neovim-config-read-only/35109/11