Open fricklerhandwerk opened 5 months ago
I agree. But I fail to understand "produce redundant test jobs". NixOS tests are not under .#checks
, but they are under .#hydraJobs
. The lines you are referring to is just the code that exports NixOS tests under .#hydraJobs
. Only the mechanism to collect them should be changed to use projects/*
directly instead of passthrus, but we need to export them somewhere, right?
Just for reference: https://github.com/ngi-nix/ngipkgs/issues/28 is about the introduction of NixOS tests. I initially wanted to put them in .#checks
, the argument against it is that "nix check
should be fast". Then there was no standard output for NixOS tests and we opted for passthrus, as this is closer to nixpkgs. (Note that I am not arguing against the issue at hand, just trying to give some context.)
Yes. I suggest .#nixosTests
, as in Nixpkgs.
Opting for passthru.test
is probably not closer to Nixpkgs, since according to pkgs/README.md
,
nixosTests.some-project
are for NixOS module tests, which spawn NixOS VMs;some-package.passthru.tests
are for NixOS package tests, which do not.While there's indeed a section on
It is purely optional, packages like Guix don't follow that.
People only do that because in Nixpkgs, ofborg does @ofborg build some-package some-package.passthru.tests
by default, and if one doesn't link module tests to the package, they'll have to manually invoke @ofborg test some-project
for them, so that makes a difference. But this doesn't apply here.
While there's indeed a section on
- Linking module tests to a package (How does the wording even make sense? It's basically pretending that a module test is a package test),
Yup, and before we had /projects
it made somewhat sense to attach NixOS tests to the "main" package for a project. Anyway, I'm all for it :)
Problem
The code in
flake.nix
is tying knots, which makes it more mysterious than necessary:https://github.com/ngi-nix/ngipkgs/blob/425f4bcd379c803e4562dc4643b8bbddf9079bfe/flake.nix#L83-L93
and also happens to produce redundant test jobs:
https://github.com/ngi-nix/ngipkgs/blob/425f4bcd379c803e4562dc4643b8bbddf9079bfe/flake.nix#L212-L223
The value of binding NixOS tests to derivation builds is questionable, as in this example
https://github.com/ngi-nix/ngipkgs/blob/425f4bcd379c803e4562dc4643b8bbddf9079bfe/pkgs/by-name/kbin/package.nix#L13-L15
because we're collecting all of them for the top-level projects anyway. And we're also primarily interested in "projects" working as a whole, which almost always requires complex test setups that aren't provided upstream, and have little to do with "the derivation builds".
Proposal
nixosTests
from package recipe inputspassthru.tests