Add feature: standalone package export (nixpkgs compatibility)

[DavHau]

The most efficient method of using dream2nix in nixpkgs would probably be to just copy the whole framework into nixpkgs and let individual packages reference the code from there. But this only makes sense if we have a stable API, otherwise we break tons of packages on each change.

In the mean time I would like to allow for another way of using dream2nix in nixpkgs. The idea of this feature is to allow users to generate standalone packages via dream2nix. These packages would consist of a dream-lock.json file plus some nix expression. That nix expression should be minimal and only require nixpkgs as an input, nothing else.

Implementation

We somehow need to know which code/files of dream2nix we need to export and put it alongside the generated dream-lock.json. This will usually be the nix file of the builder + it's dependencies inside dream2nix + some glue code, executing the builder. This requires some kind of mechanism to detect which other files of dream2nix the builder depends on.

import via symlink architecture

One idea cold be to introduce a policy, so that builders are only allowed to depend on code that is located in the same directory as the builder's default.nix. This way we always know which files we need to ship. For other files of the framework required by the builder, we can create a symlink from the builders directory to that file.

The problem is just that builders depend on externals and dlib which are both quite large and we don't want to copy those in full.

dlib

We could re-factor dlib, so that each set of independent functions lives in their own file, maximizing the amount of files, and minimizing code per file. Alongside the builders, we can then create symlinks pointing to the few files we depend on and import them from there.

externals

Looking at externalPaths in our flake.nix, we already know which ones are the relevant files to install for each external. Now we just need to learn which externals are used by a builder. For this we could add a flag externalsUsed to each bilder, signaling the need for an external.

@yusdacra I plan to start working on that soon. Let me know in case you have any ideas.

[yusdacra]

I think we can do this in a way that wouldn't require us to symlink files around. We can split dlib as you said, and we can also make each of those a module.

We can modify importModule so that after the module is imported, instead of returning the module directly it returns the path of the module and the module itself:

{
  importModule =
    {file, extraArgs}:
    let
      # stuff
      module = (import file ({inherit lib config;} // extraArgs));
    in
      {inherit module file;};
}

We can then export every module we want to expose to all modules in a big attrset:

{
  # dlib functions
  dlib-func1 = importModule {file = ...;};
  dlib-func2 = ...;
  # builders
  builders-rust-crane = ...;
  # translators
  translators-rust-cargo-lock = ...;
}

of course these wouldn't be declared by hand as we can just collect files from a directory and create an attrset (we already do for subsystems). The naming for modules would need to be unique of course.

Then again in importModule, we can do something like this now:

{
  # all modules
  importedModules = /* the attrset of modules imported with importModule */;
  allModules = l.mapAttrs (imported: imported.module) importedModules;

  importModule =
    {file, extraArgs}:
    let
      # stuff
      args = {inherit lib;} // allModules // extraArgs;
      moduleFunc = import file;
      module = moduleFunc args;
      dependsOn = let
          modulesUsed = l.attrKeys (l.functionArgs moduleFunc);
          deps = l.catAttrs modulesUsed modules;
          depsFiles = l.map (attrs: attrs.file) deps;
        in
          depsFiles ++ [file];
    in
      {inherit module file dependsOn;};
}

what this basically does is, it passes all modules to every module. Then, using the function argument names (for the module we are importing), it gets a list of all modules for those and maps the modules to their paths. We export this in a dependsOn attribute while returning, which will be the list of (module) paths a module depends on.

For a builder, this could look like this:

# a builder module, say it was under subsystems/rust/builders/crane
{dlib-func1, dlib-func2, ...}: {
  type = "pure";
  build = {...}: {...}: ...;
}

which is basically the same as what we have now, except that we don't take dlib in module args, instead we take the modules we want to use, in this case dlib-func1 and dlib-func2. The imported module for this builder would look something like this:

{
  module = {
    type = "pure";
    build = {...}: {...}: ...;
  };
  file = <the file path of this module>;
  dependsOn = [
    <the path of this module>
    <the path of dlib-func1>
    <the path of dlib-func2>
  ];
}

then we can just copy all the files there in a folder (maybe called lib) and the dream-lock.json next to that folder. The default.nix file (or whatever) would then need to have some code that looks like this to import modules:

# nixpkgs
{lib, callPackage, ...}:
let
  modules = {
    builders-rust-crane = importModule ./lib/builder;
    dlib-func1 = importModule ./lib/dlib-func1;
    dlib-func2 = importModule ./lib/dlib-func2;
  };
  importModule = file: import file (modules // {inherit lib;});
in

and then the modules can be used. Of course the whole modules stuff can just be replaced with a JSON file, then the default.nix can be generic and would just be a builtins.fromJSON and mapping over the module paths and importing them. But I don't think that matters here.

What do you think about this? One downside is you won't be able to use {...}@args syntax on modules, because if you use stuff from args those won't be found with functionArgs. But I don't think that really matters.

[yusdacra]

BTW, the above also applies to if we switch to nixos modules. It would be even easier with nixos modules since you can just export the imports as some other config attribute (eg. dependencies). It would even be merged automatically if you are using multiple builders etc.!

[DavHau]

Now that we have most of the things migrated to nixos modules, the standalone export feature is something that we could potentially tackle soon.