search

NixOS / nix

Nix, the purely functional package manager
https://nixos.org/
GNU Lesser General Public License v2.1
12.88k stars 1.53k forks source link

Give Nix deeper understanding of tests vs. builders #874

Open copumpkin opened 8 years ago

copumpkin commented 8 years ago

At the most basic level, I think we could set up the derivation primitive to take two scripts instead of one. nixpkgs's stdenv.mkDerivation wrapper could abstract over that and still present the same interface, except it could shove checkPhase and friends into the second script, possibly going out of its way to preserve the environment and cwd from the first script if it proves helpful.

The advantage to doing this is that we could start coming up with cleaner rules for tests vs. builds. Both would still affect the derivation hash, but:

  1. Tests could work with a different sandbox profile than builds. They might get more permissive network access (could introduce nondeterminism in results, but might not be a big deal), and might also not be allowed to write to $out or other outputs.
  2. Hydra and our builder tooling could present test results independently, and also offer to rerun test suites without rerunning builds
  3. We could also have different rules around #873
Ericson2314 commented 8 years ago

I'd love it if certain "seeds" for test environments could be remembered, so as to better reproduce ephemerally-failing test. For example, in the /dev/{,u}random case, the device files could be logged, so those drawn values could be "prepended" to the device file when the test is run again.

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

stale[bot] commented 2 years ago

I closed this issue due to inactivity. → More info