darwin: go: cycle detected in the references of ... (a private go package)

[mstone]

Overview

I was just bit by the same issue as #10599 while building a private go package on Darwin and as a result, I'd like to document my debugging efforts so that others who encounter similar problems may find my workaround between now and whenever we decide how to fix the issue for real.

Issue description

The symptom of the issue was that while running nix-build -K -I nixpkgs=~/nixpkgs -j2 --cores 2 shell.nix with a recent checkout of nixpkgs (https://github.com/NixOS/nixpkgs/commit/5120af001f2bb163b58c41c84b11a0c136a207fe) on a nix-expression using buildGoPackage to define a multiple-output derivation, I got a cyclic reference error like this one:

cycle detected in the references of ‘/nix/store/43wwl5kgh0z9yd92ydc7jahy1yjipmwk-go1.6-whist-20160816-ca1fec0-bin’

The fix, as in #10599, is to use install_name_tool to repoint the offending LC_RPATH tag.

Diagnosis

To diagnose the cause(s) of the problem, I rebuilt nix-1.11.3 with a custom backport of https://github.com/NixOS/nix/pull/925/commits/c87a56f4d01442ee94c0db9e89a89292d44015ae (temporarily published at https://github.com/mstone/nix/commit/a715d9ebe419d170d956dd743d072de016ffc24b) which gave me the following clue:

cycle detected in the references of ‘/nix/store/43wwl5kgh0z9yd92ydc7jahy1yjipmwk-go1.6-whist-20160816-ca1fec0-bin’ from ‘/nix/store/baswj95p801zlfhjxarran4xdyjgz4g8-go1.6-whist-20160816-ca1fec0’

Using this clue, I cd'ed to the build directory I kept with nix-build -K and grepped env-vars for the values of out= and bin= and then, using the $bin directory, I ran grep -r baswj95p801zlfhjxarran4xdyjgz4g8 /nix/store/43wwl5kgh0z9yd92ydc7jahy1yjipmwk-go1.6-whist-20160816-ca1fec0-bin to find out which files contained cyclic references, which produced one output:

Binary file /nix/store/43wwl5kgh0z9yd92ydc7jahy1yjipmwk-go1.6-whist-20160816-ca1fec0-bin/bin/.whist-wrapped matches

Next, to find out why this file contained a cyclic reference, I fiddled with otool until I ran otool -l /nix/store/43wwl5kgh0z9yd92ydc7jahy1yjipmwk-go1.6-whist-20160816-ca1fec0-bin/bin/.whist-wrapped | grep -C 5 baswj95p801zlfhjxarran4xdyjgz4g8 which indicated that the problem was an inappropriate LC_RPATH setting:

Load command 16
          cmd LC_RPATH
      cmdsize 96
         path /nix/store/baswj95p801zlfhjxarran4xdyjgz4g8-go1.6-whist-20160816-ca1fec0/lib (offset 12)

just as in #10599.

Finally, to fix the problem locally, I applied the fixes proposed in 86dae7017364cbf6cfb40f0ccd1d6b0aa7cdd933 and 4e1fdadd022ed9a1f7517b60b704c48bc30aaa75 to my own nix expression and, thankfully, they worked just fine for me once I adapted them to repoint LC_RPATH from $lib to $bin (since buildGoPackage produces derivations with outputs named out and bin, but not lib).

/cc @pikajude @vcunat @ttuegel

[vcunat]

@mstone: so you think some of the general frameworks is causing this on Darwin and should be improved?

[grahamc]

I was also struggling with this problem, and after very carefully reading and understanding what @mstone is saying was able to apply this issue to a package I'm building too. Example package using this fix:

{ pkgs ? import <nixpkgs> {} }:
let
  inherit (pkgs) buildGoPackage fetchFromGitHub darwin fixDarwinDylibNames;
in
buildGoPackage rec {
  name = "sudolikeaboss-${version}";
  version = "0.2.1";

  goPackagePath = "github.com/ravenac95/sudolikeaboss";
  src = fetchFromGitHub {
    owner = "ravenac95";
    repo = "sudolikeaboss";
    rev = "v${version}";
    sha256 = "1zsmy67d334nax76sq0g2sczp4zi19d94d3xfwgadzk7sxvw1z0m";
  };

  propagatedBuildInputs = with darwin.apple_sdk.frameworks; [
    Cocoa
    fixDarwinDylibNames
  ];

  postInstall = ''
    install_name_tool -delete_rpath $out/lib -add_rpath $bin $bin/bin/sudolikeaboss
  '';
  goDeps = ./deps.nix;
}

Thank you so much, @mstone, for so thoroughly documenting how you debugged the issue and solved the problem.

[ben0x539]

Hi y'all I ran into this with a private package too.

It feels like buildGoPackage should name its outputs dev and out instead of out and bin?

[Hodapp87]

I just ran into this on https://github.com/NixOS/nixpkgs/pull/25103

[roastiek]

I have done a little research. The $out/lib rpath is added by stdenv through NIX_LDFLAGS env. variable. However, this path is removed on Linux when shrinking rpath with patchelf. But there is no equivalent of this for Darwin.

I'm using another solution by adding NIX_NO_SELF_RPATH = true; into a derivation to turn off the default behavior of stdenv. I think this could be added to goBuildPackage safely because I don't see a way the $out/lib path could exist in a standard go derivations.

[LnL7]

@deedrah Indeed, this doesn't happen on linux because the binary is patched after the build with patchelf --shrink-rpath. Since libraries are generally not linked that way on darwin we can get rid of the rpath entries (not just for go packages). https://github.com/NixOS/nixpkgs/pull/30150

[copumpkin]

@mstone this should be fixed now that #30150 is merged. Let us know if you still see anything like this crop up!