Add `lib` output for attributes that don’t rely on `pkgs`

[sellout]

Is your feature request related to a problem? Please describe.

I often want to take advantage of filterCargoSources, etc. in a context where I have to jump through hoops to get pkgs. Sometimes I jump through the hoops, other times I hack around them with something like

filterCargoSources = import "${crane}/lib/filterCargoSources.nix" {inherit (nixpkgs) lib;};

Describe the solution you'd like

It would be great if cargo.lib contained the subset of cargo.mkLib pkgs that doesn’t need pkgs.

[ipetkov]

Hi @sellout thanks for the report!

I often want to take advantage of filterCargoSources, etc. in a context where I have to jump through hoops to get pkgs.

The general intention is that you should not be passing crane (the flake input) around, but rather craneLib (the result of crane.mkLib pkgs somewhere at the "entry-point" of your code). Would that solve the problem you are running into?

---

Although it sounds like a neat idea, I'm not sure we can truly support the notion of crane.lib which has functions that don't need a nixpkgs instantiation, because it would end up being an empty set in practice. Currently filterCargoSources requires pkgs.lib, so effectively speaking, it still requires a valid pkgs instantiation (the fact that nixpkgs doesn't come with a lib that you can consume independently is a whole separate discussion beyond our scope here).

Even if we could express filterCargoSources in a way without requiring pkgs or pkgs.lib, it would also make it a lot more cumbersome to change the code in a backwards compatible way (namely moving a function out of the hypothetical crane.lib.foo to (crane.mkLib pkgs).foo), hence why I'm doubtful this is something we can achieve in a practical sense!

[sellout]

I'm not sure we can truly support the notion of crane.lib which has functions that don't need a nixpkgs instantiation, because it would end up being an empty set in practice. Currently filterCargoSources requires pkgs.lib, so effectively speaking, it still requires a valid pkgs instantiation (the fact that nixpkgs doesn't come with a lib that you can consume independently is a whole separate discussion beyond our scope here).

Nixpkgs does provide a lib independent of pkgs. I’ve added the following to a project of mine in ./lib/crane.nix:

{
  crane,
  lib,
}: let
  internalCrateNameFromCargoToml =
    import "${crane}/lib/internalCrateNameFromCargoToml.nix" {inherit lib;};
in {
  crateNameFromCargoToml =
    import "${crane}/lib/crateNameFromCargoToml.nix" {inherit internalCrateNameFromCargoToml lib;};

  filterCargoSources = import "${crane}/lib/filterCargoSources.nix" {inherit lib;};
}

and in the flake,

lib.crane = import ./lib/crane.nix {
  inherit crane;
  inherit (nixpkgs) lib;
}

The general intention is that you should not be passing crane (the flake input) around, but rather craneLib (the result of crane.mkLib pkgs somewhere at the "entry-point" of your code). Would that solve the problem you are running into?

The issue is that the entry-point doesn’t have access to pkgs (and can’t, because that code is used in places that take different pkgs). So any terms that want to use filterCargoSources, etc. need to become functions that take pkgs.

The less frustrating cases are contexts like overlays and system-specific outputs like packages and checks, where you just need to pass in final or pkgs to those terms that are now functions. But then there are also use cases where pkgs is even less accessible – e.g., you want to define your own lib attributes that use things like filterCargoSources internally, and those are now forced to take pkgs as well, despite using nothing from it (this is the one that motivated me to open this).

And I understand not wanting to have Nixpkgs as a flake input. It’s big. I’ve been considering extracting a nixpkgs-lib flake from Nixpkgs that extracts just its lib (which would still be re-exported by Nixpkgs) to make it lighter weight.

Even if we could express filterCargoSources in a way without requiring pkgs or pkgs.lib, it would also make it a lot more cumbersome to change the code in a backwards compatible way (namely moving a function out of the hypothetical crane.lib.foo to (crane.mkLib pkgs).foo), hence why I'm doubtful this is something we can achieve in a practical sense!

I think crane.mkLib pkgs would also include the “pure” lib, like how Nixpkgs’ pkgs.lib includes everything from nixpkgs.lib. Removing a function from crane.lib, would still break some things, but it shouldn’t affect use cases where you do have pkgs, because you should prefer the more comprehensive crane.mkLib pkgs in that context.

[ipetkov]

The less frustrating cases are contexts like overlays and system-specific outputs like packages and checks, where you just need to pass in final or pkgs to those terms that are now functions. But then there are also use cases where pkgs is even less accessible – e.g., you want to define your own lib attributes that use things like filterCargoSources internally, and those are now forced to take pkgs as well, despite using nothing from it (this is the one that motivated me to open this).

Curious to understand the architecture/design of this approach. Sure you can abstract things like filterCargoSources within a common "pure" lib which doesn't care what the final packages are doing; but since you are using crane, ultimately there's going to be a call to buildPackage (or equivalent) which ostensibly does care about the exact pkgs being used (for cross compiling, etc.). How is that handled wrt to the different pkgs being juggled and why it isn't a problem there but it is for source filtering?

---

I think crane.mkLib pkgs would also include the “pure” lib, like how Nixpkgs’ pkgs.lib includes everything from nixpkgs.lib. Removing a function from crane.lib, would still break some things, but it shouldn’t affect use cases where you do have pkgs, because you should prefer the more comprehensive crane.mkLib pkgs in that context.

The thing I'm ultimately worried about is suddenly realizing that crane.pure.foo actually wants to say create a derivation or otherwise access pkgs. Now we have to deprecate it in favor of craneLib.foo which is annoying to consumers (esp ones who got used to using it in "pure" contexts). If that ends up feeling too onerous, then the option is to define craneLib.fooButBetter (or worse craneLib.foo) and dealing with the confusion which which one ought to be used.