Usage with Nix Flakes

[shadowrylander]

Hello!

I was just wondering how I would go about using this with Nix Flakes, as in what module I would have to import in nixpkgs.lib.nixosSystem.modules? Would mach-nix.nixosModules.mach-nix work?

Thank you kindly for the help!

[DavHau]

Mach-nix does currently not propose a module. Also I'm not sure what would be the benefit. Mach-nix is just a package. There is nothing that needs to be integrated with the system.

[shadowrylander]

Ah; could you show me how to use it in a flake for the NixOS Configuration, then? I'm a bit confused on how to pass it to the config; do I just add it to the list of arguments I pass to nixpkgs.lib.nixosSystem?

[DavHau]

A nixos configuration with the mach-nix cmdline tool inside the system environment could look like this:

{ pkgs, ... }:
let
  mach-nix =  (import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix.git";
    ref = "refs/heads/master" ;
    rev ="70b6328ca28b54a135a19d8d7267a3d1639c3b3d";
  })).mach-nix;
in
{
  environment.systemPackages = [
    mach-nix
  ];
}

[shadowrylander]

But what if I were to include it in the inputs set in flake.nix? How would the respective output set look?

[DavHau]

Maybe it helps to play around with it in nix repl a bit:

nix-repl> f = (builtins.getFlake "github:davhau/mach-nix")     

nix-repl> f.packages.x86_64-linux.mach-nix
«derivation /nix/store/a9wgylch2j0j59bf7blbcnm4323zi86f-python3.7-mach-nix-2.4.1.drv»

[shadowrylander]

Yeah, I'm still kinda' stuck on where to put [f].packages.x86_64-linux.mach-nix in flake.nix...

[DavHau]

Sorry, I was busy finishing the 3.0.0 release. Were you able to solve your problem in the meantime?

[shadowrylander]

No problem! I think the problem was actually solved by the PR I sent; now I'm wondering how to include mach-nix in the list of modules sent to my configuration file. I can't seem to find that anywhere in the NixOS / Nix Flakes documentation! I really need to help with that, actually...

[DavHau]

Not sure what you actually try to achieve, but wouldn't it be the easiest way to just import a mach-nix expression directly into your config? Flakes are still experimental. Maybe it's better to not use it?

[shadowrylander]

I don't want to keep manually updating the module rev and hash, though... It's why I use the master branch everywhere; I wanted to copy a rolling-release model. Flakes seemed to finally be the easiest way to do it.

[shadowrylander]

Actually, I figured I'll just pass mach-nix directly to my overlays as an argument, so problem solved! More or less... I eagerly await your opinion on the matter!

[shadowrylander]

Turns out I get stuck on attribute 'mkPython' missing when using the current mach-nix flake and passing mach-nix.packages.${system}.mach-nix. This what I'm getting:

unpacking channels...
building the system configuration...
warning: updating lock file '/etc/nixos/flake.lock':
* Updated 'mach-nix': 'path:/etc/nixos/extras/mach-nix?narHash=sha256-Omm0wC5aBAKag7Ev4TupSmTNAYDvZgx9BwoNcyawhMA=' -> 'github:davhau/mach-nix/8db91624a6413aefe142d126e749b38b725087fd'
error: --- EvalError --- nix
at: (12:19) in file: /nix/store/ikrq860lflwqcbw0jiick0pfd25y05db-source/shared/global/xonsh.nix

    11|       # build a mach-nix python env
    12|       machnixPy = mach-nix.mkPython {
      |                   ^
    13|         packagesExtra = [

attribute 'mkPython' missing
---- show-trace ----
trace: while evaluating the attribute 'xonsh'
at: (91:37) in file: /nix/store/ikrq860lflwqcbw0jiick0pfd25y05db-source/shared/global/xonsh.nix

    90|       # make a final overlay with the modified xonsh
    91|       finalOverlay = self: super: { xonsh = xonsh_with_pkgs; };
      |                                     ^
    92|     in

trace: while evaluating anonymous lambda
at: (1:1) in file: /nix/store/ikrq860lflwqcbw0jiick0pfd25y05db-source/shared/global/xonsh.nix

     1| {
      | ^
     2|   pkgs,

trace: from call site
at: (31:33) in file: /nix/store/ikrq860lflwqcbw0jiick0pfd25y05db-source/shared/global/_nixpkgs.nix

    30|         (self: super: { qtile = import ./qtile.nix base.mach-nix; })
    31|         (self: super: { xonsh = import ./xonsh.nix (base.mach-nix // { inherit lib; }); })
      |                                 ^
    32|         mozilla

trace: while evaluating the attribute 'xonsh'
at: (31:25) in file: /nix/store/ikrq860lflwqcbw0jiick0pfd25y05db-source/shared/global/_nixpkgs.nix

    30|         (self: super: { qtile = import ./qtile.nix base.mach-nix; })
    31|         (self: super: { xonsh = import ./xonsh.nix (base.mach-nix // { inherit lib; }); })
      |                         ^
    32|         mozilla

trace: while evaluating the attribute 'value.content'
at: (654:14) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/modules.nix

   653|     { _type = "override";
   654|       inherit priority content;
      |              ^
   655|     };

trace: while evaluating the attribute 'value._type'
at: (594:73) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/modules.nix

   593|       highestPrio = foldl' (prio: def: min (getPrio def) prio) 9999 defs;
   594|       strip = def: if def.value._type or "" == "override" then def // { value = def.value.content; } else def;
      |                                                                         ^
   595|     in {

trace: while evaluating anonymous lambda
at: (493:19) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/modules.nix

   492|           # Avoid sorting if we don't have to.
   493|           if any (def: def.value._type or "" == "order") defs''.values
      |                   ^
   494|           then sortProperties defs''.values

trace: from call site
at: (493:14) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/modules.nix

   492|           # Avoid sorting if we don't have to.
   493|           if any (def: def.value._type or "" == "order") defs''.values
      |              ^
   494|           then sortProperties defs''.values

trace: while evaluating the attribute 'values'
at: (497:9) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/modules.nix

   496|       in {
   497|         values = defs''';
      |         ^
   498|         inherit (defs'') highestPrio;

trace: while evaluating the attribute 'mergedValue'
at: (503:5) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/modules.nix

   502|     # Type-check the remaining definitions, and merge them. Or throw if no definitions.
   503|     mergedValue =
      |     ^
   504|       if isDefined then

trace: while evaluating the option `users.users.root.shell':
trace: while evaluating the attribute 'value'
at: (471:9) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/modules.nix

   470|     in warnDeprecation opt //
   471|       { value = builtins.addErrorContext "while evaluating the option `${showOption loc}':" value;
      |         ^
   472|         inherit (res.defsFinal') highestPrio;

trace: while evaluating anonymous lambda
at: (98:72) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/modules.nix

    97|           # For definitions that have an associated option
    98|           declaredConfig = mapAttrsRecursiveCond (v: ! isOption v) (_: v: v.value) options;
      |                                                                        ^
    99| 

trace: from call site
at: (279:20) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/attrsets.nix

   278|               then recurse (path ++ [name]) value
   279|               else f (path ++ [name]) value;
      |                    ^
   280|         in mapAttrs g set;

trace: while evaluating 'g'
at: (276:19) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/attrsets.nix

   275|           g =
   276|             name: value:
      |                   ^
   277|             if isAttrs value && cond value

trace: from call site
trace: while evaluating the attribute 'shell'
trace: while evaluating 'isDerivation'
at: (305:18) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/attrsets.nix

   304|   */
   305|   isDerivation = x: isAttrs x && x ? type && x.type == "derivation";
      |                  ^
   306| 

trace: from call site
at: (280:18) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/types.nix

   279|       name = "package";
   280|       check = x: isDerivation x || isStorePath x;
      |                  ^
   281|       merge = loc: defs:

trace: while evaluating 'check'
at: (280:15) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/types.nix

   279|       name = "package";
   280|       check = x: isDerivation x || isStorePath x;
      |               ^
   281|       merge = loc: defs:

trace: from call site
at: (287:19) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/types.nix

   286|     shellPackage = package // {
   287|       check = x: (package.check x) && (hasAttr "shellPath" x);
      |                   ^
   288|     };

trace: while evaluating 'check'
at: (287:15) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/types.nix

   286|     shellPackage = package // {
   287|       check = x: (package.check x) && (hasAttr "shellPath" x);
      |               ^
   288|     };

trace: from call site
at: (28:8) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/nixos/lib/utils.nix

    27|   toShellPath = shell:
    28|     if types.shellPackage.check shell then
      |        ^
    29|       "/run/current-system/sw${shell.shellPath}"

trace: while evaluating 'toShellPath'
at: (27:17) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/nixos/lib/utils.nix

    26|   # Returns a system path for a given shell package
    27|   toShellPath = shell:
      |                 ^
    28|     if types.shellPackage.check shell then

trace: from call site
at: (399:17) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/nixos/modules/config/users-groups.nix

   398|           initialPassword initialHashedPassword;
   399|         shell = utils.toShellPath u.shell;
      |                 ^
   400|       }) cfg.users;

trace: while evaluating the attribute 'text' of the derivation 'users-groups.json'
at: (7:7) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/pkgs/build-support/trivial-builders.nix

     6|     stdenv.mkDerivation ({
     7|       name = lib.strings.sanitizeDerivationName name;
      |       ^
     8|       inherit buildCommand;

trace: while evaluating the attribute 'text'
at: (77:38) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/strings-with-deps.nix

    76| 
    77|   stringAfter = deps: text: { inherit text deps; };
      |                                      ^
    78| 

trace: while evaluating the attribute 'text'
at: (9:5) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/nixos/modules/system/activation/activation-script.nix

     8|   addAttributeName = mapAttrs (a: v: v // {
     9|     text = ''
      |     ^
    10|       #### Activation script snippet ${a}:

trace: while evaluating 'id'
at: (14:5) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/trivial.nix

    13|     # The value to return
    14|     x: x;
      |     ^
    15| 

trace: from call site
trace: while evaluating 'textClosureMap'
at: (70:35) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/lib/strings-with-deps.nix

    69| 
    70|   textClosureMap = f: predefined: names:
      |                                   ^
    71|     concatStringsSep "\n" (map f (textClosureList predefined names));

trace: from call site
at: (89:18) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/nixos/modules/system/activation/activation-script.nix

    88|                 withHeadlines = addAttributeName set';
    89|               in textClosureMap id (withHeadlines) (attrNames withHeadlines)
      |                  ^
    90|             }

trace: while evaluating the attribute 'system.activationScripts.script'
at: (68:9) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/nixos/modules/system/activation/activation-script.nix

    67|       apply = set: {
    68|         script =
      |         ^
    69|           ''

trace: while evaluating the attribute 'activationScript' of the derivation 'nixos-system-siluam-21.03.20201011.9e51c25'
at: (95:5) in file: /nix/store/ih6jlh451lf0whq63pxgcb2wy8yajsxm-source/nixos/modules/system/activation/top-level.nix

    94|   baseSystem = pkgs.stdenvNoCC.mkDerivation {
    95|     name = "nixos-system-${config.system.name}-${config.system.nixos.label}";
      |     ^
    96|     preferLocalBuild = true;

This is my flake.nix:

{
  description = "shadowrylander";

  inputs = rec {
    home-manager = {
      url = "github:nix-community/home-manager/master";
      # https://github.com/nix-community/home-manager/blob/master/flake.nix#L4
      # HM takes 'nixpkgs' as input
      inputs.nixpkgs.follows = "nixpkgs";
    };
    mach-nix = {
        url = "github:davhau/mach-nix/master";
        # url = "/home/shadowrylander/mach-nix";
        inputs.nixpkgs.follows = "nixpkgs";
    };
    impermanence = {
      url = "github:nix-community/impermanence";
      flake = false;
    };
    flake-utils = {
        url = "github:numtide/flake-utils";
        inputs.nixpkgs.follows = "nixpkgs";
    };
    nur = {
      url = github:nix-community/NUR;
      inputs.nixpkgs.follows = "nixpkgs";
    };

    r2003.url = "github:NixOS/nixpkgs/nixos-20.03";
    r2009.url = "github:NixOS/nixpkgs/nixos-20.09";
    unstable.url = "github:NixOS/nixpkgs/nixos-unstable";
    small.url = "github:NixOS/nixpkgs/nixos-unstable-small";
    master.url = "github:NixOS/nixpkgs/master";

    nixpkgs.follows = "master";

    nix = { url = "github:NixOS/nix/master"; };

  };

  outputs = inputs@{ self, nixpkgs, flake-utils, ... }: let

    stdenvs = (import (./. + "${global}/stdenvs.nix") {
      inherit nixpkgs;
    });

    systems = {
      imd = let _imd = stdenvs.imd; in {
        x64 = _imd.x64.system;
        x86 = _imd.x86.system;
      };
      arm = let _arm = stdenvs.arm; in {
        x64 = _arm.x64.system;
        x86 = _arm.x86.system;
      };
    };
    system = systems.imd.x64;

    r2003 = final: prev: { r2003 = inputs.r2003.legacyPackages.${system}; };
    r2009 = final: prev: { r2009 = inputs.r2009.legacyPackages.${system}; };
    unstable = final: prev: { unstable = inputs.unstable.legacyPackages.${system}; };
    small = final: prev: { small = inputs.small.legacyPackages.${system}; };

    global = "/shared/global";
    _pkgs = (import nixpkgs {});
    __pkgs = _: (import nixpkgs _);
    lib = _pkgs.lib;

    pkgs = __pkgs (import (./. + "${global}/_nixpkgs.nix") {
      inherit lib;

      ###############################################
      mach-nix = inputs.mach-nix.packages.${system}.mach-nix;
      ###############################################

      stdenv = {
        inherit system;
        package = _pkgs.gcc10.stdenv;
      };
      pkgs = _pkgs;
    });

  in {
    overlays = [ r2003 r2009 unstable small inputs.nur.overlay ];

    nixosConfigurations = let

      base = {
        global = {
          modules = with inputs; [
            home-manager.nixosModules.home-manager
            # impermanence
          ];
        };
        create = { inherit system pkgs; };
      };

      create = { hostname, system, pkgs }: let
        config = ./. + "/configs/${hostname}.nix";
      in nixpkgs.lib.nixosSystem {
        inherit system pkgs;
        modules = base.global.modules ++ [ (import config) ];
      };

    in {
      siluam = create (base.create // { hostname = "siluam"; });
    };
  };
}

This is my _nixpkgs.nix:

{
    stdenv,
    pkgs,
    lib,
    # mach-nix ? (import (builtins.fetchGit {
    #     url = "https://github.com/DavHau/mach-nix/";
    #     ref = "master";
    # }) { inherit pkgs; python = "python38"; }),
    mach-nix,
    ...
}:

let
    mozilla = import (builtins.fetchTarball https://github.com/mozilla/nixpkgs-mozilla/archive/master.tar.gz);
    base = {
        mach-nix = { inherit pkgs mach-nix; };
    };
in {
    config = {
        allowUnfree = true;
        allowBroken = true;
        allowUnsupportedSystem = true;
        # preBuild = ''
        #     makeFlagsArray+=(CFLAGS="-w")
        #     buildFlagsArray+=(CC=cc)
        # '';
        buildEnv.override = { inherit stdenv; };
    };
    overlays = [
        (self: super: { qtile = import ./qtile.nix base.mach-nix; })
        (self: super: { xonsh = import ./xonsh.nix (base.mach-nix // { inherit lib; }); })
        mozilla
    ];
}

And this is xonsh.nix:

{
  pkgs,
  lib,
  mach-nix,
  ...
}:

let
  xonshOverlay = {pkgs, lib, ...}:
    let
      # build a mach-nix python env
      machnixPy = mach-nix.mkPython {
        packagesExtra = [
          # "https://github.com/psf/requests/tarball/v2.22.0"

          # "https://gitlab.com/picotech/nanotech/nanite/-/archive/master/nanite-master.tar.gz"
          (mach-nix.buildPythonPackage {
            src = builtins.fetchGit{
              url = "https://gitlab.com/picotech/nanotech/nanite";
              ref = "master";
            };
          })

          "https://files.pythonhosted.org/packages/bc/ab/c49f97516f78c2b0cacb4f45873abc4ca9872942a9c4c19ded8052c8edda/python-wifi-0.6.1.tar.bz2"
        ];
        requirements = ''
          # add requirements here
          # jedi
          borgmatic
          dephell
          ply
          prompt_toolkit
          pygments
          pipx
          yubico-client

          # add xontribs here
          # xonsh-direnv
          # xonsh-docker-tabcomplete
          # xonsh-vox-tabcomplete
          # xontrib-prompt-bar
          # xontrib-ssh-agent
          xonsh-autoxsh
          xontrib-autojump
          xontrib-fzf-widgets
          xontrib-kitty
          xontrib-pipeliner
          xontrib-powerline2
          xontrib-prompt-vi-mode
          xontrib-readable-traceback
          xontrib-schedule
          xontrib-z
        '';
      };

      # get mach-nix modified nixpkgs
      newPkgs = machnixPy.nixpkgs;

      ### modify the xonsh package
      pythonPackages = newPkgs.python38.pkgs;
      # switch xonsh to python 3.8
      xonsh_py_38 = newPkgs.xonsh.override {
        python3Packages = newPkgs.python38Packages;
      };
      # add extra packages to PYTHONPATH of xonsh
      xonsh_with_pkgs = xonsh_py_38.overrideAttrs (oa: rec {
        version = "0.9.23";

        # fetch from github because the pypi package ships incomplete tests
        src = pkgs.fetchFromGitHub {
          owner  = "xonsh";
          repo   = "xonsh";
          rev    = version;
          sha256 = "1by13ryq9ldc9wln3fk5mm6zvjp4aim57ikw49v0dfmz8irnpglp";
        };

        pythonPath =
          (if (builtins.hasAttr "pythonPath" oa) then oa.pythonPath else [])
          ++ machnixPy.selectPkgs pythonPackages
          ++ (with pythonPackages; [
            ply
            prompt_toolkit
            pygments
            pipx
            yubico-client
            nixpkgs
          ]);
      });

      # make a final overlay with the modified xonsh
      finalOverlay = self: super: { xonsh = xonsh_with_pkgs; };
    in
    finalOverlay;
in
  let
    pkgs = import (builtins.fetchGit {
      url = "https://github.com/NixOS/nixpkgs/";
      ref = "master";
    }) { config = {}; overlays = [];};
    overlay = xonshOverlay { inherit pkgs lib; };
  in
    (import pkgs.path { overlays = [ overlay ]; }).xonsh

Interestingly, this does not seem to be affecting my qtile.nix:

{
  pkgs,
  mach-nix,
  ...
}:

let
  qtileOverlay = {pkgs, ...}:
    let

      # build a mach-nix python env
      machnixPy = mach-nix.mkPython {
        packagesExtra = [
          # "https://github.com/psf/requests/tarball/v2.22.0"

          # "https://gitlab.com/picotech/nanotech/nanite/-/archive/master/nanite-master.tar.gz"
          (mach-nix.buildPythonPackage {
            src = builtins.fetchGit {
              url = "https://gitlab.com/picotech/nanotech/nanite";
              ref = "master";
            };
          })

          "https://files.pythonhosted.org/packages/bc/ab/c49f97516f78c2b0cacb4f45873abc4ca9872942a9c4c19ded8052c8edda/python-wifi-0.6.1.tar.bz2"
        ];
        requirements = ''
          iwlib
        '';
        _.iwlib.buildInputs.add = [ pkgs.wirelesstools ];
      };

      # get mach-nix modified nixpkgs
      newPkgs = machnixPy.nixpkgs;

      ### modify the qtile package
      pythonPackages = newPkgs.python38.pkgs;
      # switch qtile to python 3.8
      qtile_py_38 = newPkgs.qtile.override {
        python37Packages = newPkgs.python38Packages;
      };
      # add extra packages to PYTHONPATH of qtile
      qtile_with_pkgs = qtile_py_38.overrideAttrs (oa: {
        pythonPath =
          oa.pythonPath
          ++ machnixPy.selectPkgs pythonPackages
          ++ (with pythonPackages; [
            # From https://github.com/NixOS/nixpkgs/issues/45039
            dateutil
            dbus-python
            keyring
            mpd2
            psutil
            pyxdg
            nixpkgs
          ]);
      });

      # make a final overlay with the modified qtile
      finalOverlay = self: super: { qtile = qtile_with_pkgs; };
    in
    finalOverlay;
in
  let
    pkgs = import (builtins.fetchGit {
      url = "https://github.com/NixOS/nixpkgs/";
      ref = "master";
    }) { config = {}; overlays = [];};
    overlay = qtileOverlay { inherit pkgs; };
  in
    (import pkgs.path { overlays = [ overlay ]; }).qtile

[shadowrylander]

Pinging @DavHau! Sorry, I'm really excited for this! 😅😻

[DavHau]

I think the problem is that the mach-nix flake doesn't expose the mkPython function. You can add it here and make a PR if you like (https://github.com/DavHau/mach-nix/blob/master/flake.nix#L19-L25)

[shadowrylander]

How would I do that, though? If I'm passing mach-nix.packages.mach-nix to my module, and my module calls mach-nix.packages.mach-nix.mkPython, i.e. that module's mkPython? Modifying the flake wouldn't exactly work, would it? You'd have to modify the mach-nix attribute set in default.nix.

[SomeoneSerge]

I think the problem is that the mach-nix flake doesn't expose the mkPython function. You can add it here and make a PR if you like (https://github.com/DavHau/mach-nix/blob/master/flake.nix#L19-L25)

Hi! I've run into this too. I was thinking about exposing mkPython through the lib flake output, but so far didn't succeed (hadn't much time to investigate this yet): https://github.com/newkozlukov/mach-nix/pull/1/

To keep the graph strongly connected, I also raised the question in https://discourse.nixos.org/t/flake-nix-interface-nix-show-fails-with-aarch64-linux-required-to-build/9561 but this issue seems much more appropriate place for discussion

Thanks!

P.S. I'm still unsure if lib is the right place

[shadowrylander]

The way I'm thinking of doing it is either by passing my module all the mk functions individually, or creating a new set with all the mk functions included as attributes.

[DavHau]

@newkozlukov Thanks for looking into this! Feel free to just open a WIP PR with your changes and we can try to fix it from there.

[shadowrylander]

It seems as though none of my changes work; any help on them would be greatly appreciated!

[SomeoneSerge]

@shadowrylander Tbh, I'm pretty occupied lately and made no progress since last comment

[shadowrylander]

Aw! 😹 Ah well; whenever you get around to it, if you do!

[nat543207]

Flake inputs actually have some special handling that causes toString flake-input-name to return a Nix store path; as a result, you can import Nix files from flake inputs directly. I've been using that strategy in my own flake-based projects; a working (but very simple) example flake that does this with mach-nix is shown below:

{
  description = "Using mach-nix from a flake";

  inputs = {
    nixpkgs.url = github:NixOS/nixpkgs/nixos-unstable;
    mach-nix.url = github:DavHau/mach-nix/f80ee22; # v3.0.0 release tag
  };

  outputs = { self, nixpkgs, mach-nix }: let
    pkgs = (import nixpkgs { system = "x86_64-linux"; }).pkgs;
    mach-nix-utils = import mach-nix {
      inherit pkgs;
      python = "python3";
    };
  in {

    packages.x86_64-linux.customPython = mach-nix-utils.mkPython {
      requirements = ''
        pytest
      '';
    };

    defaultPackage.x86_64-linux = self.packages.x86_64-linux.customPython;
  };
}

mkPython works just fine, which you can see for yourself by initializing a repo with the above flake and running

[nat543207@hostname|tmp] nix shell .#customPython                                                                                                                                                             
[nat543207@hostname|tmp] python
Python 3.8.6 (default, Sep 23 2020, 13:54:27)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytest
>>> pytest.fail('Intentional failure')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/6qbpa7k8mlgzb8gf67lms6ypx4q6vfp2-python3-3.8.6-env/lib/python3.8/site-packages/_pytest/outcomes.py", line 153, in fail
    raise Failed(msg=msg, pytrace=pytrace)
Failed: Intentional failure
>>>

However, functions like buildPythonApplication that rely on extractor don't run correctly for some arguments (e. g. URLs) due to a <nixpkgs> import that can't be evaluated in pure-mode by the flake. ~I've got a local patch to replace that import with the pkgs value used to initialize mach-nix; it hasn't caused me any issues yet, so I'll see if I can put a PR up for it tomorrow.~ This can be worked around by providing pname, version, and requirements manually in the argument attrset, which causes the logic that depends on the extractor to be skipped. My local patch doesn't actually resolve this issue, and it looks like doing so would require digging deeper into the code than I have time for right now.

[shadowrylander]

If I may ask something slightly off-topic: if anyone here has experience extending the default lib module, could you help me over on reddit with a major roadblock I'm having? It seems nobody else has the necessary time or experience to help me there on this particular matter.

Afterwards, I can continue helping debug the mach-nix flake, as the aforementioned issue seems to be stopping me from doing so.

[shadowrylander]

I can confirm that @nat543207's method works.

[shadowrylander]

I suppose I'll close this then!

[DavHau]

I merged some commits of @nat543207 and @newkozlukov and added the library functions to flakes.nix.

[ae-mo]

Hey, I've tried the approach @nat543207 suggested, but nix develop fails with the following error output:

...
building '/nix/store/gpc8k09imm8yki7xzi9kx8c2c3m5fbi4-python3.8-scs-2.1.2.drv'...
error: --- Error --- nix-daemon
builder for '/nix/store/gpc8k09imm8yki7xzi9kx8c2c3m5fbi4-python3.8-scs-2.1.2.drv' failed with exit code 1; last 10 log lines:
  ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.3/lib'
  ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.18.1/lib'
  ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.33.0/lib'
  ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-6.3p08/lib'
  ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.2/lib'
  ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.1.1h/lib'
  ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-osx-10.8.5/lib'
  ld: framework not found Accelerate
  clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
  error: Command "clang -bundle -undefined dynamic_lookup -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.11/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.6.0.1/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.2.10/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.5/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.3/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.18.1/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.33.0/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-6.3p08/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.1.1h/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-osx-10.8.5/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.11/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.6.0.1/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.2.10/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.5/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.3/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.18.1/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.33.0/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-6.3p08/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.1.1h/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-osx-10.8.5/lib build/temp.macosx-10.6-x86_64-3.8/src/scsmodule.o build/temp.macosx-10.6-x86_64-3.8/scs/src/util.o build/temp.macosx-10.6-x86_64-3.8/scs/src/linalg.o build/temp.macosx-10.6-x86_64-3.8/scs/src/scs_version.o build/temp.macosx-10.6-x86_64-3.8/scs/src/cones.o build/temp.macosx-10.6-x86_64-3.8/scs/src/ctrlc.o build/temp.macosx-10.6-x86_64-3.8/scs/src/aa.o build/temp.macosx-10.6-x86_64-3.8/scs/src/normalize.o build/temp.macosx-10.6-x86_64-3.8/scs/src/scs.o build/temp.macosx-10.6-x86_64-3.8/scs/src/rw.o build/temp.macosx-10.6-x86_64-3.8/scs/linsys/common.o -L/nix/store/0mabx9absir73mmijrsv0dsyvyya463r-python3-3.8.6/lib -o build/lib.macosx-10.6-x86_64-3.8/_scs_direct.cpython-38-darwin.so -Wl,-framework -Wl,Accelerate -Wl,-framework -Wl,Accelerate" failed with exit status 1
error: --- Error --- nix-daemon
1 dependencies of derivation '/nix/store/xzj8zdqali8vk9xb6mzic3igrw57awwj-python3-3.8.6-env.drv' failed to build
error: --- Error -------------------------------------------------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/brszc18sx6k91x4l1dcxr87i8nmzb0cq-mach-nix-python-shell-env.drv' failed to build

My flake.nix:

{
  description = "Flake for portmanteau - portfolio manager to end all uncertainties";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    mach-nix = {
      url = github:DavHau/mach-nix/1ec92303acd142aa1a3b60bb97745544cf049312; # v3.1.1 release tag
      inputs.nixpkgs.follows = "nixpkgs";
      inputs.flake-utils.follows = "flake-utils";
    };
  };
  outputs = { self, nixpkgs, mach-nix, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = (import nixpkgs { inherit system; }).pkgs;
        mach-nix-utils = import mach-nix {
          inherit pkgs;
          python = "python3";
        };
        requirements = ''
            pyportfolioopt
            pytest
          '';
      in rec
      {

        devShell = mach-nix-utils.mkPythonShell {
          inherit requirements;
        };

        packages.portmanteau = mach-nix-utils.mkPython {
          inherit requirements;
        };

        defaultPackage = packages.portmanteau;

      }
  );
}

I'm on macOS. Any ideas?

[shadowrylander]

@ae-mo You might want to check on that error; it seems to be screaming. 😹

[ae-mo]

It seems to be a problem related to the scs python package, not to the flake itself, so I guess this is not the place to discuss it 😄

[DavHau]

@ae-mo you could try settings the provider for scs to nixpkgs. Or try with the 'conda-beta' brach of mach-nix which can take packages from conda.

[ae-mo]

@DavHau I switched to Linux in the end and it worked, after fiddling with providers. Thanks!

[SomeoneSerge]

Run into this again and it wasn't immediately obvious to me how to use this, so posting the snippet here (as probably the easiest googlable place).

TLDR is flake-utils.eachSystem transforms lib into outputs.lib.${system}, so one should currently use e.g. mach-nix.lib.${system}.mkPython.

Note that poetry2nix takes a different approach and puts their convenience functions into overlays instead. This possibly makes sense, as lib is (probably; better ask Nix maintainers) supposed to be a place for system-independent outputs:

• poetry2nix:

  • flake.nix
  • the output that has mk* functions: overlay.nix, accessible via pkgs in your nix files

• flake-utils just puts a flat top-level lib

• • *

    Snippet that currently works

Step 0:

# flake.nix
{
  description = "...";

  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.mach-nix.url = "github:DavHau/mach-nix";
  inputs.mach-nix.inputs.nixpkgs.follows = "nixpkgs";

  outputs = { self, nixpkgs, flake-utils, mach-nix }: (flake-utils.lib.eachDefaultSystem (system:
  let
    pkgs = import nixpkgs { inherit system; };
  in {
    devShell = (import ./shell.nix { nixpkgs=pkgs; mach-nix=mach-nix.lib.${system}; });
  }));
}

Step 1:

# shell.nix

{ mach-nix, nixpkgs ? (import <nixpkgs> { })}:

with nixpkgs;

let
  myEnv = mach-nix.mkPythonShell {
    requirements = ''
      opencv-python
      numpy
      msgpack
      asyncio-nats-client
      '';
  };
in myEnv

Step 2: Happily run nix develop

[DavHau]

@SomeoneSerge Thanks for noticing this. Currently the lib functions of mach-nix are in fact system dependent. They all return a derivation and that derivation is built via a certain nixpkgs. We could remove lib's dependency on nixpkgs, and instead make nixpkgs a mandatory argument whenever you call one of the lib functions. But then you would ignore the nixpkgs specified via your flakes.lock. Would you really want that?

[SomeoneSerge]

@DavHau Hmm, if nixpkgs is made an argument of mach-nix.lib.mk*, then it seems they all would get the client flake's nixpkgs input (pinned in user's flake.lock) opposed to inputs.mach-nix.inputs.nixpkgs (pinned in mach-nix's flake.lock) - doesn't look criminal to me.

Does anything in mk* assume specific nixpkgs revisions, like attribute names? What we could lose is that right now mach-nix's flake.lock provides a combination of (mach-nix revision, nixpkgs revision) that is guaranteed/expected to work, whereas taking nixpkgs from the user means that when the user runs her build the first time (before her flake.lock is generated) it may possibly fail