numtide / nix-filter

a small self-contained source filtering lib
MIT License
200 stars 17 forks source link

Can this be used to filter the contents of a flake's input? #34

Closed nothingnesses closed 1 year ago

nothingnesses commented 1 year ago

I'm trying to make a flake.nix for cargo-generate. The following works:

{
  description = "cargo, make me a project";
  inputs = {
    cargo-generate = {
      flake = false;
      url = "github:cargo-generate/cargo-generate";
    };
    nci = {
      inputs.nixpkgs.follows = "nixpkgs";
      url = "github:yusdacra/nix-cargo-integration";
    };
    nix-filter.url = "github:numtide/nix-filter";
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };
  outputs = inputs:
    let
      name = "cargo-generate";
      nix-filter = import inputs.nix-filter;
      pkgs = common: packages:
        builtins.map (element: common.pkgs.${element}) packages;
    in inputs.nci.lib.makeOutputs {
      config = common: {
        cCompiler = { package = common.pkgs.clang; };
        outputs = {
          defaults = {
            app = name;
            package = name;
          };
        };
        runtimeLibs = pkgs common [ "openssl" ];
      };
      pkgConfig = common:
        let
          override = {
            buildInputs = pkgs common [ "openssl" "perl" "pkg-config" ];
          };
        in {
          ${name} = {
            app = true;
            build = true;
            depsOverrides = { inherit override; };
            overrides = { inherit override; };
            profiles = { release = false; };
          };
        };
      root = inputs.cargo-generate.outPath;
    };
}

But it also includes some files that are unnecessary to the actual build. I tried replacing root with the following instead:

      root = nix-filter {
        root = inputs.cargo-generate.outPath;
        include = [ "./Cargo.lock" "./Cargo.toml" "./README.md" "./src" ];
      };

But this fails with "error: getting status of '/nix/store/0ccnxa25whszw7mgbgyzdm4nqc0zwnm8-source/Cargo.toml': No such file or directory" when running nix build. Listing the contents of that directory also shows that it's empty.

Is it possible to use use nix-filter in this way? If not, what would be an alternative solution?

zimbatm commented 1 year ago

I'm not sure. This might be a limitation of nix. If you look at the nix issue tracker, you'll see a bunch of issues around builtins.path and flakes.

ilkecan commented 1 year ago

root = inputs.cargo-generate.outPath;

Here inputs.cargo-generate.outPath is "/nix/store/5fr6a7r2h3axxlv83xgkdvxjlh8rh4r5-source", which is a Nix string.


Strings and paths inside include/exclude are mapped internally to transform them to a matcher function using this function: https://github.com/numtide/nix-filter/blob/1a3b735e13e90a8d2fd5629f2f8363bd7ffbbec7/default.nix#L99-L111 An example of a _toMatcher call would be _toMatcher { inherit root; } "./Cargo.lock". path_ on the 109th line above should be the normalized absolute path of "./Cargo.lock" according to the root. And this is done with _toCleanPath: https://github.com/numtide/nix-filter/blob/1a3b735e13e90a8d2fd5629f2f8363bd7ffbbec7/default.nix#L119-L129 Here the relevant line is 127 since path is a Nix string that doesn't start with a slash ("./Cargo.lock"). Because absPath is also a Nix string, concatenation happens between Nix strings and path normalization doesn't happen (i.e. ./ is not stripped away). Note that if inputs.cargo-generate.outPath was a Nix path, this wouldn't be a problem:

nix-repl> inputs.cargo-generate.outPath
"/nix/store/5fr6a7r2h3axxlv83xgkdvxjlh8rh4r5-source"

nix-repl> toString ("/nix/store/5fr6a7r2h3axxlv83xgkdvxjlh8rh4r5-source" + ("/" + "./Cargo.lock"))
"/nix/store/5fr6a7r2h3axxlv83xgkdvxjlh8rh4r5-source/./Cargo.lock"

nix-repl> toString (/nix/store/5fr6a7r2h3axxlv83xgkdvxjlh8rh4r5-source + ("/" + "./Cargo.lock"))
"/nix/store/5fr6a7r2h3axxlv83xgkdvxjlh8rh4r5-source/Cargo.lock"

In the end "/nix/store/5fr6a7r2h3axxlv83xgkdvxjlh8rh4r5-source/./Cargo.lock" is whitelisted and comparison with "/nix/store/5fr6a7r2h3axxlv83xgkdvxjlh8rh4r5-source/Cargo.lock" fails. You can do

      root = nix-filter {
        root = inputs.cargo-generate;   # `outPath` is implicit
        include = [ "Cargo.lock" "Cargo.toml" "README.md" "src" ];
      };

to prevent this.

nothingnesses commented 1 year ago

@ilkecan I can confirm, that solution works. Thanks for sharing it and for explaining why it wasn't working.