Extend `nix flake check` to support custom attributes

[NobbZ]

Is your feature request related to a problem? Please describe.

When running nix flake check one regularly sees warnings emitted by nix about unknown attributes, or default* moved to something else.

Especially the unknown attributes can be annoying, even more those that evolved into a quasi-standard, like lib (even used by nixpkgs) or homeModules and homeConfigurations.

Describe the solution you'd like

There might be yet another flake output which is not a derivation but instead a function that takes and arbitrary value and returns a list of string or set of string.

I initially wanted to use checks for this, though that is already used for something else. In lack of a better name, I therefore use nixChecks in the following examples.

Once nix flake check encounters an attribute foo it doesn't know how to check, it calls self.nixChecks.foo self.foo.

nixChecks can then use arbitrary nix functions to verify the value of foo to match the requirements and issue informative messages, warnings or errors. It should be avoided to use traceing functions from the nixChecks though it lies in the nature of the checks, that they require at least partial evaluation of the values, and therefore might trigger traceing calls within the value.

A message is either a set with level and message (where level is "info", "warning", or "error".) or alternaively just a string, which is assumed to be the message for a warning then.

Whenever at least one error got emitted, the exit code of nix flake check should be non-zero.

Describe alternatives you've considered

External programs that would check validity of custom attributes while continue to ignore warnings issued by nix flake check, or not run it at all.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/custom-flake-outputs-for-checks/18877/6

[thufschmitt]

I’m not entirely convinced since it means that each flake would need to specify its own outputs schema, which would be a pain in most cases − for example, I’d be very happy to have nix flake check stop complaining about my homeConfigurations output, but it would be sad if I had to write a custom nixCheck field just for that (even if it just means re-exporting it from the home-manager flake).

Maybe (or in addition?) nix flake check should instead just ignore the fields it doesn’t know about (most cases being false-positives anyways)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/custom-flake-outputs-for-checks/18877/21

[tomberek]

There are several similar situations where one would like the behavior of a well-known output path to be applied to another one or extended.

random brainstorm:

app2.behavesLike = "apps";
packages.behavesLike = "legacyPackages";
hydraJobs_feature_branch.behavesLike = "hydraJobs";
foo.behavesLike = {
   check = self: assert (1 == 1) true;
   build = self: packages;
   show = self: "my custom label";
};

I can see this quickly getting out of hand.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-would-you-like-to-see-improved-in-nix-cli-experience/24012/2

[cyntheticfox]

Maybe the ability to disable the warning via a config setting like warn-unknown-outputs you could set in a flake's nixConfig might help?

EDIT: Could also consider an option like custom-outputs to define known ones as others have suggested above...