nix-eval-jobs has inconsistant behaviour with Hydra

[JulienMalka]

Explanation of the bug

nix-eval-jobs has inconsistant behaviour with hydra, making it impossible to use to evaluate the nixpkgs jobsets.

As explained here, the evaluation of pkgs/top-level/release.nix will return an attribute set containing fields of this format : recurseForDerivations = { }, which is not recognized by the Nix evaluator (the recurseForDerivation attribute should be true or false and indicate if Nix should evaluate each attribute of the current attribute set).
When evaluating jobs, Hydra always recursively evaluate. When facing recurseForDerivation = { }, Hydra just doesn't care skip this attribute. On the other hand, nix-eval-jobs has support for recurseForDerivations and will not recursively evaluate by default. It will respect the recurseForDerivationfield, but it will report an error if the value is not a boolean. The flag --force-recurse is designed to have the same behavior as Hydra, that is always recurse. The bug here is that even if force-recurseis set, nix-eval-jobswith try to evaluate the value of recurseForDerivations and if the value is not a boolean fail and stop recursing, which makes it impossible to evaluate the nixpkgs/trunkjobset for example.

Example

One can try to evaluate the #hydraJobsfield of this flake :

{
  description = "A debugging flake";

  outputs = { self }:

    let
      sample = derivation { name = "sample"; builder = "sample"; system = "mysystem"; };
    in
    {
      hydraJobs = {
        one-level = sample;
        two-levels-no-recurse = {
          b = sample;
        };
        two-levels-recurse-bool = {
          b = sample;
          recurseForDerivations = true;
        };
        two-levels-recurse-set = {
          b = sample;
          recurseForDerivations = { };
        };
      };
    };
}

Here is the output of hydra-eval-jobs:

error: attribute 'two-levels-recurse-bool.recurseForDerivations' is a Boolean, which is not supported
{
  "one-level": {
    "description": "",
    "drvPath": "/nix/store/nj11brp99izzmilnsnhnq7jjnh0v25f8-sample.drv",
    "homepage": "",
    "isChannel": false,
    "license": "",
    "maintainers": "",
    "maxSilent": 7200,
    "nixName": "sample",
    "outputs": {
      "out": "/nix/store/b8yq27p8p7ck5vl869g0bgxljqzlayw6-sample"
    },
    "schedulingPriority": 100,
    "system": "mysystem",
    "timeout": 36000
  },
  "two-levels-no-recurse.b": {
    "description": "",
    "drvPath": "/nix/store/nj11brp99izzmilnsnhnq7jjnh0v25f8-sample.drv",
    "homepage": "",
    "isChannel": false,
    "license": "",
    "maintainers": "",
    "maxSilent": 7200,
    "nixName": "sample",
    "outputs": {
      "out": "/nix/store/b8yq27p8p7ck5vl869g0bgxljqzlayw6-sample"
    },
    "schedulingPriority": 100,
    "system": "mysystem",
    "timeout": 36000
  },
  "two-levels-recurse-bool.b": {
    "description": "",
    "drvPath": "/nix/store/nj11brp99izzmilnsnhnq7jjnh0v25f8-sample.drv",
    "homepage": "",
    "isChannel": false,
    "license": "",
    "maintainers": "",
    "maxSilent": 7200,
    "nixName": "sample",
    "outputs": {
      "out": "/nix/store/b8yq27p8p7ck5vl869g0bgxljqzlayw6-sample"
    },
    "schedulingPriority": 100,
    "system": "mysystem",
    "timeout": 36000
  },
  "two-levels-recurse-bool.recurseForDerivations": {
    "error": "error: attribute 'two-levels-recurse-bool.recurseForDerivations' is a Boolean, which is not supported"
  },
  "two-levels-recurse-set.b": {
    "description": "",
    "drvPath": "/nix/store/nj11brp99izzmilnsnhnq7jjnh0v25f8-sample.drv",
    "homepage": "",
    "isChannel": false,
    "license": "",
    "maintainers": "",
    "maxSilent": 7200,
    "nixName": "sample",
    "outputs": {
      "out": "/nix/store/b8yq27p8p7ck5vl869g0bgxljqzlayw6-sample"
    },
    "schedulingPriority": 100,
    "system": "mysystem",
    "timeout": 36000
  }
}

Here is the result of nix-eval-jobs --force-recurse:

{"attr":"one-level","attrPath":["one-level"],"drvPath":"/nix/store/nj11brp99izzmilnsnhnq7jjnh0v25f8-sample.drv","name":"sample","outputs":{"out":"/nix/store/b8yq27p8p7ck5vl869g0bgxljqzlayw6-sample"},"system":"mysystem"}
{"attr":"two-levels-no-recurse.b","attrPath":["two-levels-no-recurse","b"],"drvPath":"/nix/store/nj11brp99izzmilnsnhnq7jjnh0v25f8-sample.drv","name":"sample","outputs":{"out":"/nix/store/b8yq27p8p7ck5vl869g0bgxljqzlayw6-sample"},"system":"mysystem"}
{"attr":"two-levels-recurse-bool.b","attrPath":["two-levels-recurse-bool","b"],"drvPath":"/nix/store/nj11brp99izzmilnsnhnq7jjnh0v25f8-sample.drv","name":"sample","outputs":{"out":"/nix/store/b8yq27p8p7ck5vl869g0bgxljqzlayw6-sample"},"system":"mysystem"}
error: value is a set while a Boolean was expected

       at /nix/store/d6726bxp3j496yp8yygfahhfwn41pwdg-source/eval-exps/flake.nix:21:11:

           20|           b = sample;
           21|           recurseForDerivations = { };
             |           ^
           22|         };
{"attr":"two-levels-recurse-set","attrPath":["two-levels-recurse-set"],"error":"error: value is a set while a Boolean was expected\n\n       at /nix/store/d6726bxp3j496yp8yygfahhfwn41pwdg-source/eval-exps/flake.nix:21:11:\n\n           20|           b = sample;\n           21|           recurseForDerivations = { };\n             |           ^\n           22|         };"}```