home-manager like module system

[Gerschtli]

Hey,

I want to come up with a solution for configuring nix-on-droid with or without home-manager. I came up with multiple solutions and want to hear your opinions about it, before I implement any of them.

I prototyped a minimal version, where I am able to set options in ~/.config/nixpkgs/config.nix like this:

{
  nix-on-droid = {
    # entry point for nix-on-droid module system
    test = "abc";
  };
}

Note: It's possible and my preferred option to define a separate config file like ~/.config/nixpkgs/nix-on-droid.nix. This way we get better error messages (prints path to file) and we have similar config files to the ones in nixos and home-manager. Example:

# entry point for nix-on-droid module system
{
  test = "abc";
}

The implementation of this module could look like this:

{ config, lib, pkgs, ... }:

with lib;

{
  options = {
    test = mkOption {
      type = types.str;
      default = "default value test";
      description = "Internal option";
    };

    test2 = mkOption {
      type = types.str;
      default = "default value test2";
      description = "Internal option2";
    };
  };

  config = {
    nix-on-droid.packages = [
      (pkgs.writeTextDir "testfile" ''
        test: ${config.test}
        test2: ${config.test2}
      '')
    ];
  };
}

With this way we could replace the static list of packages in default.nix in basic-environment with the merged nix-on-droid.packages list. The nix-on-droid-linker has to be called after every change and needs to remove old links.

Instead of the nix-on-droid.packages list and the fragile nix-on-droid-linker setup, we could provide an executable similar to home-manager script.

The only results of possible nix-on-droid options and configs I can think of is managing /data/data/com.termux.nix/files/usr directory. Therefore I recommend implementing something like environment.etc in nixos (https://github.com/NixOS/nixpkgs/tree/master/nixos/modules/system/etc) for just this directory. This should be fairly easy to implement, provides atomic upgrades and removes old files. With this implementation we also don't need this basic-environment anymore.

Concerning the integration with home-manager, refactoring the home.file option for our case could be a possible solution but this would mean we have to support both ways which are conflicting a bit. The better solution would be, if we could inject our activation script logic (linking etc like environment.etc) in the one of home-manager, but currently there isn't any option. We would need something like that: https://github.com/rycee/home-manager/pull/779.

I would say it's save enough to try the custom module system and custom nix-on-droid activator script and ignore the home-manager integration for now. home-manager would still work perfectly fine with this solution, the user just has to manage to config files until we get a better solution.

What do you say?

[t184256]

(discussion that doesn't involve managing /etc:)

I prototyped a minimal version, where I am able to set options in ~/.config/nixpkgs/config.nix like this:

Great. Do I understand correctly, that, if one chooses to do everything through home-manager, there's an prexisting facility to provide the desired values through nixpkgs.config and a pure h-m setup is still controllable through h-m?

Note: It's possible and my preferred option to define a separate config file like ~/.config/nixpkgs/nix-on-droid.nix. This way we get better error messages (prints path to file) and we have similar config files to the ones in nixos and home-manager.

This will make the user manage three files and I don't see the benefits. Can you provide an example of an unreadable traceback that you're trying to get rid of?

---

(managing /etc:)

With this way we could replace the static list of packages in default.nix in basic-environment with the merged nix-on-droid.packages list.

The static (part of the) list should cover just the login script and the dependencies, right? It sounds maintainable to keep it static.

Regarding management of /etc, I engaged into some brainstorming and outlined the options that I currently see:

• Stay where we are (not cool, /etc is not easily extensible).

• Devise a mechanism to override what lives inside what's currently basic-environment and plumb it into h-m if necessary, where configuration might happen: 1) through ~/.config/nixpkgs/config.nix, replaced by some h-m mechanism if user installs h-m 2) through ~/.config/nixpkgs/config.nix, even if user installs h-m 3) through a separate file 4) through h-m config exclusively and symlinking into /etc (activation) might happen: a) through our system, hooked into home-manager's activation mechanism b) through our system, and we wrap home-manager's invocations c) through our system, and they have to be invoked separately d) through h-m, by extending it to manage files outside of ~ (I vaguely recall @rycee being unusually receptive to that idea, but my memory might be playing tricks on me) e) through h-m, by tricking it into thinking that /etc is inside ~ f) through h-m, by just symlinking /etc to ~/.profile/etc (with the added bonus of getting more meaningful extra files in /etc for free)

• Build a separate mechanism for injecting more files into /etc, with pretty much the same configuration and activation choices.

And now it's high time to discuss our expectations for the setup without h-m. My original intention for it was 'execute as little as possible on-device and drop an experienced user into a nix-powered shell 1) to minimize the chance of failure of the initial setup 2) to impose as little as possible onto the power user'. We've deviated a lot since then, and now I'm fine with what we have, with the only support guarantee for h-m-less being 'if you do nothing but update, you can still log it'. I'm OK with leaving everything, even parametrizing basic-environment, to the user. I'm also fine with not caring about scenarios where the user installs h-m and chooses to bypass it (e.g., with nix-env).

Given that, my current preferences are: override>separate>stay, 1>4>2>3, and f>d>e>a>b>c.

Your thoughts?

[Gerschtli]

Great. Do I understand correctly, that, if one chooses to do everything through home-manager, there's an prexisting facility to provide the desired values through nixpkgs.config and a pure h-m setup is still controllable through h-m?

I don't understand, what you are trying to ask. What is the "pre-existing facility" and what do you mean with "desired values"? h-m would be completely unaffected by any changes in this setup with the only exception that it should not install basic-environment.

This will make the user manage three files and I don't see the benefits. Can you provide an example of an unreadable traceback that you're trying to get rid of?

It doesn't work either way.. There has to be another reason causing messages like error: The optiontestx' defined in <unknown-file>' does not exist. But just my 2 cents: I think it would be better if every file (config.nix, home.nix, nix-on-droid.nix) has only one purpose. Nesting a complete module system into config.nix doesn't feel right. Anyway I will try to fix the problem with the error message and if it works both ways, I will follow your decision :)

And now it's high time to discuss our expectations for the setup without h-m.

I think that is the point where we should start. I'm skipping your extensive list of options because I think you and me are heading in different directions.

My original intention for it was 'execute as little as possible on-device and drop an experienced user into a nix-powered shell 1) to minimize the chance of failure of the initial setup 2) to impose as little as possible onto the power user'.

I'm totally fine with this and it's my goal too, but what I had in mind with this whole idea was to not only provide some rudimentary basic functionality for h-m-less setup. I wanted to provide the full feature-set or at least as far as I can get to a similar integration with h-m. Because if we go the h-m first approach, the chances are very low that anyone continues with the h-m-less way. I try to unify both the h-m-less and the h-m setup as much as I can to reduce development/maintenance effort and deliver the most functionality possible.

If we follow one of your preferences of f>d>e, we would need to refactor h-m logic and still provide a way to manage h-m-less setups (or stay with the fragile nix-on-droid-linker thing). In the option a we could reuse that logic and the user doesn't need to call a possible nix-on-droid script to update the configs. Options b and c don't seem reasonable to me too, too much indirection for the user.

f) through h-m, by just symlinking /etc to ~/.profile/etc (with the added bonus of getting more meaningful extra files in /etc for free)

BTW: This could have unwanted side effects. I'm not sure why we should link complete ~/.profile/etc. I like the approach of preferring explicit over implicit.

[Gerschtli]

It doesn't work either way.. There has to be another reason causing messages like error: The option testx' defined in ' does not exist.

I found the issue, silly mistake on my side..

Excerpt of module system:

let
  homeDir = builtins.getEnv "HOME";
  configFile = homeDir + "/.config/nixpkgs/nix-on-droid.nix";

  rawModule = evalModules {
    modules = [
      {
        _module.args.pkgs = pkgs;
      }
    ]
    # option with config.nix
    ++ optional (pkgs.config ? nix-on-droid) pkgs.config.nix-on-droid
    # option with nix-on-droid.nix
    ++ optional (builtins.pathExists configFile) configFile
    ++ import ./module-list.nix;
  };
in
...

config.nix:

{
  nix-on-droid = {
    testx = "blaaa";
  };
}

Produces: The option `testx' defined in `<unknown-file>' does not exist.

nix-on-droid.nix:

{
  testx = "blaaa";
}

Produces: The option `testx' defined in `/home/tobias/.config/nixpkgs/nix-on-droid.nix' does not exist.

[t184256]

I don't understand, what you are trying to ask.

Can one do `nixpkgs.config.nix-on-droid.test = "abc"; from h-m and get it picked up?

I think it would be better if every file (config.nix, home.nix, nix-on-droid.nix) has only one purpose.

Can't argue with that. But choice between splitting and not splitting is even better, and it seems like config.nix is already optionally controllable from home.nix (for h-m management intents and purposes).

Because if we go the h-m first approach, the chances are very low that anyone continues with the h-m-less way.

True.

I try to unify both the h-m-less and the h-m setup as much as I can to reduce development/maintenance effort and deliver the most functionality possible.

And I'm fine with landing your improvements. I don't oppose h-m-less considerations just because they're not what I use, and I'm OK with paying a reasonable price of complexity for them. Tying stuff to h-m is not a goal for me (or I would've done that long ago). I just don't want to also test h-m-less setups, especially the upgrade path. Makes me think that I should finally learn to test in i686 VMs.

BTW: This could have unwanted side effects. I'm not sure why we should link complete ~/.profile/etc. I like the approach of preferring explicit over implicit.

Unwanted side effects and positive side effects, lots of them. Yeah, that's a potentially very dangerous decision that is undoable without breaking user setups. But it's so tempting at times...

That being said, if we limit the discussion to 'h-m-less setups should also work', then yeah, f,d,e,4 become quite unreasonable, we seem to have an agreement on b and c being undesirable, and we're left with:

1) through ~/.config/nixpkgs/config.nix, replaced by some h-m mechanism if user installs h-m 2) through ~/.config/nixpkgs/config.nix, even if user installs h-m 3) through a separate file

x

a) through our system, hooked into home-manager's activation mechanism

I'm pretty sure that 'a' is unavoidable here, or h-m would not exist in its form today =) Maybe there's also

g) through our system, and h-m hooks into it the way it hooks into NixOS configuration

That sounds complicated, but maybe this aligns more with what you're trying to build, does it?

Personally, I'd like to avoid it being unconfigurable from a single config file, but both a h-m config or our config that integrates h-m as a module sound fine.

[Gerschtli]

Can one do `nixpkgs.config.nix-on-droid.test = "abc"; from h-m and get it picked up?

The value would be set for this exact instantiation of nixpkgs. If you want to persist it, you need to write the config.nix file by yourself (via h-m for example ;) )

Can't argue with that. But choice between splitting and not splitting is even better, and it seems like config.nix is already optionally controllable from home.nix (for h-m management intents and purposes).

Yes, of course. We can provide both ways as option, why not? :) But I don't think there is a option in h-m to write config.nix other than `xdg.configFile."nixpkgs/config.nix".

I just don't want to also test h-m-less setups, especially the upgrade path.

I feel you, but I think it worth it. It is just one more test. Would love to automate some of the manual testing but don't have a suitable idea for that..

g) through our system, and h-m hooks into it the way it hooks into NixOS configuration That sounds complicated, but maybe this aligns more with what you're trying to build, does it?

I don't think that this should be overly complicated. Just have a look at nix-darwin/default.nix which seems to be a pretty good starting point for nix-on-droid. You are right, this is the way I want it to go but didn't come up with this idea. The reason why I'm into this way is that the installation should be managed by us and h-m should be optionally added on top of it. If we use home-manager and home.nix directly, we need to merge our setup into it, which is the completely wrong order. It would be comparable of managing your nixos system with h-m and integrating your nixos configuration into your home.nix.

That way there is no question between

1. through ~/.config/nixpkgs/config.nix, replaced by some h-m mechanism if user installs h-m
2. through ~/.config/nixpkgs/config.nix, even if user installs h-m
3. through a separate file

because you would add your home.nix config as an option in our config file like:

{
  nix-on-droid = {
    test = "abc"; # option of nix-on-droid
    homeFile = ./home.nix; # link to home-manager
  };
}

On top of that, we could provide a h-m module to set some more options like the glibc env variable etc.

Result of all that would be:

• Complete setup can be controlled via one config file (with and without h-m) -> Totally agnostic whether h-m or other module systems are pulled in.
• We don't need to refactor the activation script and home.file logic in h-m.
• The user has nix-on-droid config installed per default and does not need to replace or move it when adding h-m.
• We need to write our on linking logic for files in /data/data/com.termux.nix/files/usr and write a nix-on-droid script similar to home-manager.
• We do not necessarily have to test multiple upgrade paths as we are always using our own.

Did I miss something? Sounds pretty sophisticated to me.

[t184256]

But I don't think there is a option in h-m to write config.nix other than xdg.configFile."nixpkgs/config.nix".

There's nixpkgs.config to control the same stuff from within h-m, even if the scope is a little different. Generating config files as text files and then reevaluating them as an input, consequently, splitting a switch into several switches, is not the way to go.

more options like the glibc env variable etc.

I didn't get it. What is that and why it explicitly belongs to h-m?

Overall, we seem to be converging on something. If the end result

• can configure /etc
• is manageable through one file
• can manage h-m through the same file as well (similarly to it can be configured from NixOS configuration)
• provides a user with a single command to rebuild current environment, in sync and in one step
• requires no explicit extra installation steps

I see little more left to be desired except for reasonable simplicity of implementation and not bricking existing installations entirely.

And it's hard to estimate the complexity for me. I know that we will end up reimplementing stuff that's already duplicated across NixOS, h-m and nix-darwin, but I still don't know how much stuff is that. But it seems like you find fun in learning exactly this, so who am I to stop you =D

[Gerschtli]

There's nixpkgs.config to control the same stuff from within h-m, even if the scope is a little different. Generating config files as text files and then reevaluating them as an input, consequently, splitting a switch into several switches, is not the way to go.

Yes, I know this option but it does only work in global way if you do the following like stated in the manual, but as this does not really relate to this current issue, let's leave it this way.

{
  nixpkgs.config = import ./nixpkgs-config.nix;
  xdg.configFile."nixpkgs/config.nix".source = ./nixpkgs-config.nix;
}

I didn't get it. What is that and why it explicitly belongs to h-m?

Sorry for the confusion. What I meant was this excerpt of our default home.nix:

{
  # Fix locale (perl apps panic without it)
  home.sessionVariables = {
    LOCALE_ARCHIVE = "${pkgs.glibcLocales}/lib/locale/locale-archive";
  };
}

This should belong in a h-m module as that would definitely add a lot complexity into our nix-on-droid script because we would need to add shell startup script managing into it. And as the setup is usable without it, I would not bother with it for now. Maybe later :)

Overall, we seem to be converging on something.

It seems so and I'm looking forward in building some new functionality. But I'm not really sure, how I am able to avoid breaking existing installations or how to provide some upgrade mechanisms but I try my very best. Maybe I get an idea while implementing it. At least I hope this will be the last big breaking change, everything else should be manageable with a stateVersion.

It does not seem to be that much to reimplement. Things like defining options, reading config, assertions, warnings and building a simple activation script works totally fine already with fewer code than I expected. Shouldn't be that hard to build the remaining pieces.