Module system: "jig"

[roberth]

Description

Don't we all want better control over modules, and better performance?

In this issue, I propose a function which I think is feasible to implement: mkJig

A device in manufacturing, woodworking, or other creative endeavors for controlling the location, path of movement, or both of either a workpiece or the tool that is operating upon it. (wiktionary)

This would be a function that, given some settings, transforms a module into a module.

mkJig :: { ... } -> Module -> Module
         ^^^^^^^ some settings
                    ^^^^^^ passed module
                              ^^^^^^ returned module

The settings restrict the structure of the returned module, and perhaps also restrict or alter the module arguments available to the passed module.

Doing this is ambitious. This function is effectively an alternate (simpler?) evalModules that functions as a proxy instead of computing its own fixpoint.

This kind of transformation has the potential to solve a number of issues:

• Loading rarely used modules lazily. Only the jig is loaded, but it can cut evaluation short, because the interesting bits weren't needed. Consider for example:

  mkJig {
  inputs = ["services" "postgresql"];
  outputs = ["systemd.services"];
  } ./postgres.nix

  This would translate to a module that declares options.services.postgresql to be a submodule with options forwarded from ./postgres.nix's options.services.postgresql. A definition config.systemd.services = mkIf options.services.postgresql.isDefined x where x comes from ./postgres.nix's config. As long as no definitions are given for options.services.postgresql, there's no reason to load postgres.nix.

  • Ideally services.postgresql wouldn't have to be a submodule, but any option tree. However, isDefined is currently only available for actual options; not for all option trees.
  • This is particularly valuable if postgres.nix comes from a yet to be fetched source, such as a flake input.

• Restricting "side effects". As you can see in the example above, mkJig won't let postgres.nix affect environment.etc.

• Altering the environment. mkJig could have its own specialArgs argument, replacing https://github.com/NixOS/nixpkgs/pull/230588 in a way that the args are automatically forwarded to imported modules: nicer DX

  • Make sure duplicate imports are caught!

• Replace partitions which currently require separate evalModules fixpoints for each "use case" or partition. This can be inefficient.

Caveats

• Increased complexity in the module system
• Overhead unknown
• No isDefined for option trees yet

Context

• The potential performance improvement overlaps a little with https://github.com/NixOS/nixpkgs/pull/267111, but that PR also solves the problem of having multiple instances of a service, and it is a more elegant solution because it models the domain better with existing means, instead of further complicating the module system.

Notify maintainers

@infinisil @roberth

---

---

Add a :+1: reaction to issues you find important.

[Aleksanaa]

I thought so before, but then the to-do list was too long so I didn’t raise this issue again.

It is easy to design a fancy new module system, but what we should think about is how to design one that is good enough but also allows us to easily migrate from the current state (just requiring some simple tidying up work or even automatic treewide changes).

Also CC @GTrunSec who did a lot of experiment on module system related works.

[roberth]

how to design one that is good enough but also allows us to easily migrate from the current state

Absolutely.

I think you're responding to:

This function is effectively an alternate (simpler?) evalModules that functions as a proxy instead of computing its own fixpoint.

Not the best phrasing of mine. It won't be a types type or an actual alternative to evalModules; just an implementation of something that has a resemblance to evalModules. The goal is for it to produce modules that are compatible with evalModules.