nix-prefetch-git determinism problems

[sbourdeauducq]

I would like to see two changes to Hydra:

• update the nix-prefetch-git script, which is currently a copy of a 2011 version with only minor changes. The new version (in release 18.09) allows for deterministic output with .git present, and is also compatible with fetchgit. I have a patch here: https://github.com/m-labs/hydra/commit/b9295c9690c32bb9698aeadf7de031b22d87677f The only thing I changed from the original nix-prefetch-git is echo "$hash" at the end. I think this can be merged upstream - it is a bit hacky, but not worse than what is currently in Hydra.
• allow setting NIX_PREFETCH_GIT_LEAVE_DOT_GIT independently from NIX_PREFETCH_GIT_DEEP_CLONE. The former allows for deterministic output but the latter does not. I don't know Perl or how to implement this without breaking compatibility, so I simply hardcoded what I wanted into the Hydra we're using here: https://github.com/m-labs/hydra/commit/b652be47f61a5241de43d80e4ff376e71fb09584

Background and rationale for those changes: we are using Hydra for continuous build, testing and distribution of a (complex) Python package, and I found all existing options to be deficient for different reasons:

• the default channel uses synthesized channel expressions based on mkFakeDerivation. The fake derivations only work with nix-env and cannot be passed into python3.withPackages (or any of the other Python environment building functions). This would prevent our Python software from being installed in a Python environment alongside other user-defined Python packages, which is not acceptable.
• defining a channel copies the nix expressions from the repository verbatim. Since this is a CI system, our package uses a relative path as the src attribute of the main derivation. On the channel subscriber, this src attribute points to a completely bogus path (e.g. /home/sb/.nix-defexpr/channels/m-labs/../../.) and renders the package uninstallable.
• the solution I implemented is to inject a synthesized src attribute based on fetchgit in the channel expressions (https://github.com/m-labs/nix-scripts/blob/31a82dd8358db95bab3be970955697fbb28c99a5/main.nix#L4). This bumps into the Hydra problems above.
• injecting a /nix/store entry instead of fetchgit does not work because while nix-env -i /nix/store/... downloads the entry if it does not exist locally, all other Nix invokations do not implement this behavior and return an error instead. This renders the package uninstallable.

If anyone else needs those modifications, here is the configuration.nix entry:

  services.hydra = {
    package = pkgs.hydra.overrideAttrs (oldAttrs: {
      src = pkgs.fetchFromGitHub {
        owner = "m-labs";
        repo = "hydra";
        rev = "b652be47f61a5241de43d80e4ff376e71fb09584";
        sha256 = "0gk4r8p2vc9vm9fxp3hckrhfzsqv39p2w6vlzkg2fly8b06pb2sl";
      };
    });
....
  }

[MaxGabriel]

@sbourdeauducq Can I ask how you're setting NIX_PREFETCH_GIT_LEAVE_DOT_GIT? The extraEnv setting hasn't worked for me and I resorted to copying across the postInstall and setting it there. Wondering if you had a better way.

postInstall = ''
              mkdir -p $out/nix-support
              for i in $out/bin/*; do
                  read -n 4 chars < $i
                  if [[ $chars =~ ELF ]]; then continue; fi
                  wrapProgram $i \
                      --prefix PERL5LIB ':' $out/libexec/hydra/lib:$PERL5LIB \
                      --prefix PATH ':' $out/bin:$hydraPath \
                      --set HYDRA_RELEASE ${version} \
                      --set HYDRA_HOME $out/libexec/hydra \
                      --set NIX_RELEASE ${nix.name or "unknown"}
                      --set NIX_PREFETCH_GIT_LEAVE_DOT_GIT "1"
              done
            ''; # */

[sbourdeauducq]

@MaxGabriel This is all inside Hydra - in the Perl source code. I'm not sure if you can set those options externally with wrapProgram, I just apply patches to Hydra (e.g. https://github.com/m-labs/hydra/commit/b652be47f61a5241de43d80e4ff376e71fb09584).