hydraJobs accepts function arguments

[edolstra]

Reported in https://github.com/NixOS/nix/issues/7031. hydra-eval-jobs apparently allows hydraJobs to accept function arguments, which is not intended, since that breaks hermeticity/reproducibility.

I think the Hydra web interfaces hides the ability to set input arguments when the jobset type is set to "flakes".

[shlevy]

How does it break hermeticity? The function args are passed in by hydra and are tracked as part of the jobset. You can look at a jobset and reevaluate it exactly as it was evaluated.

As I noted in https://github.com/NixOS/nix/issues/7031, without this you can't actually use hydra for continuous integration. That is a major regression in flake mode!

[knedlsepp]

I also still don't get how hydra will ever be able to gain back the "CI" feature with flakes. I envision it to work along these lines: https://github.com/NixOS/nix/issues/5673 But there was still no feedback if that is the solution that will be implemented at some point. (I guess it will be addressed at some point as in a couple of presentations @edolstra already acknowledged this problem and that he "just did not come around to implementing it", which gives me the impression that there must be some kind of "master plan" on how this should be done)

[edolstra]

How does it break hermeticity?

Because it accepts arguments from outside of the flake. So if a user does nix build flake#hydraJobs.<foo>, they'll get a different result from what Hydra produced.

[shlevy]

What if we had something like:

{
  inputs = {
    a.url = "a";
    b.url = "b";
  };

  outputs = { a, b, self }@inputs: let
    jobsetFun = { a ? inputs.a , b ? inputs.b }: {
      c = a.makeC { inherit b; src = self; };
    };
  in {
    hydraJobsets = {
      default = {
        type = "hydra-jobset";
        jobs = jobsetFun {};
      };

      integrate = {
        type = "hydra-jobset";
        inputOverrides = {
          a.url = "github:NixOS/a/develop";
          b.update = true;
          d.url = "d";
        };
        jobs = { a, b, d, self }: jobsetFun {
          inherit b;
          a = d.tweakA a;
        };
      };
    };
  };
}

With the semantics that a hydra jobset pointing to integrate would create a new lockfile based on inputOverrides (including updating b to the latest even though its flakeref is unchanged) and use that to evaluate jobs? Then we could have that updated lockfile associated with the jobset and fetchable by the user, and the user could either evaluate locally in the same re-locking mode or evaluate locally with the re-locking already supplied by hydra.

Would this be acceptable?

[jbgi]

Less versatile than function arguments but still useful and does not break hermeticity: https://github.com/NixOS/hydra/pull/1143