Undocumented / impossible: override go modules

[doronbehar]

Describe the bug

I'm trying to write an overlay that will override attributes of a package that is built with buildGoModule and I'm having trouble overriding the modules derivation of it.

It seems the author's intention was that I'd use overrideModAttrs as an argument to buildGoModule? But I guess this doesn't work with overlays? Whatever I do, I get either an evaluation error or a build error because the old go-modules is used. Here are a few examples I tried:

overridedPackage = super.original.overrideAttrs(oldAttrs: rec {
  src = ....;
  modSha256 = "0000000000000000000000000000000000000000000000000000";

But the build still took the old go modules derivation.

I also tried instead:

  overrideModAttrs = (_oldAttrs : {
    modSha256 = "0r062nka72ah2nb2gf8dfrrj4sxadkykcqjzkp4c9vwk93mhw41k";
  });

Which failed with:

error: cannot coerce a function to a string, at ...

And I even went as far as:

  configurePhase = (let
    go-modules = self.pistol.go-modules.overrideAttrs(oldAttrs: {
      modSha256 = "0000000000000000000000000000000000000000000000000000";
    });
  in
    ''
      echo checking I override configure phase
      runHook preConfigure

      export GOCACHE=$TMPDIR/go-cache
      export GOPATH="$TMPDIR/go"
      export GOSUMDB=off
      export GOPROXY=file://${go-modules}

      cd "$modRoot"

      runHook postConfigure
    ''
  );
});

But that also failed because the build used the old go-modules. I can confirm though, that it did override the configurePhase which is the only place ${go-modules} is evaluated.

Expected behavior

It shouldn't be that difficult to override a go modules derivation.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context

I'm trying to override a package I maintain which it's update is stale (https://github.com/NixOS/nixpkgs/pull/85452), due to an unresolved issue with the version build flags :(.

Notify maintainers

@kalbasit ? As the author of buildGoModule according to git blame...

[kalbasit]

I have the same issue, unfortunately, and I have not been able to look into it.

[kalbasit]

@doronbehar this worked for me

  (self: super: {
    bazel-gazelle = (super.callPackage "${super.path}/pkgs/development/tools/bazel-gazelle" {
      buildGoModule = args: super.buildGoModule (args // {
        modSha256 = "13yx70p0x7hpdp2zn7zrl0i88fk6dhcryp0lmn0zwdn92ldm3n5y";
        patches = (args.patches or []) ++ super.lib.lists.singleton (super.fetchpatch {
          url = "https://github.com/bazelbuild/bazel-gazelle/pull/749.patch";
          sha256 = "03sxbgwvb6hxwpmk9mckwjzgxank9difc0bgvzl6x3rir76ivwmj";
        });
      });
    });
  })

[doronbehar]

Damn you are overriding the goddamn function. It's nice to know this is not "Impossible". Do you think @kalbasit this interface could be improved somehow?

[doronbehar]

@kalbasit I've stumbled upon another point to improve the golang docs at https://github.com/NixOS/nixpkgs/pull/97856 . I'm not sure whether this should be documented or not along with the other changes - overrideModAttrs doesn't seem to be able to provide what it promises.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/inconsistent-vendoring-in-buildgomodule-when-overriding-source/9225/4

[joulaud]

The problem with pythonBuildPackage described at https://discourse.nixos.org/t/un-disabling-a-python-package-in-an-overlay/10593 seems similar to me. Perhaps overrideAttrs documentation should be improved to better explain those traps.

[fiksn]

I've also stumbled on this when trying to override some go package which involved changing vendorSha256 too. So if I understood this correctly the problem with overrideAttrs is that it works just with a simple mkDerivation, but buildGoModule internally does more and calls mkDerivation also for the vendored dependencies?

Then I suppose in the same way as there is overridePythonAttrs there should be some overrideGoAttrs.

[doronbehar]

Then I suppose in the same way as there is overridePythonAttrs there should be some overrideGoAttrs.

I think we should strive to have less of such functions with different names but the same meaning.

[Mic92]

This one is a bit shorter:

(pkgs.istioctl.overrideAttrs (old: let
  version = "1.7.1";
  src = pkgs.fetchFromGitHub {
    owner = "istio";
    repo = "istio";
    rev = version;
    sha256 = "sha256-CvYqYxb19lSc61SrvGcf/kG9e9zosHz4JbByA+bN600=";
  };
in rec {
  name = "istioct-${version}";
  inherit src;
  go-modules = (pkgs.buildGoModule {
    inherit name src;
    vendorSha256 = "sha256-4Z4Fgv9zmAwp3tEvHj8yLOWkFY/zFz5VfehSWCuIUcI=";
  }).go-modules;
}))

[nh2]

This one is a bit shorter:

@Mic92 this doesn't work for me, trying with:

{
  kopia_unstable = nixos-2105.kopia.overrideAttrs (old:
    let
      version = "ede09c15854ce74fde40603a6857ed3991ba666c";
      src = nixos-2105.fetchFromGitHub {
        owner = "kopia";
        repo = "kopia";
        rev = version;
        sha256 = "0wdkwpsgmycbiq8dvng3d0ps5qp87b7k1xvw8rv2wk1qzrrp6d93";
      };
    in
      rec {
        name = "kopia-${version}";
        inherit src;
        go-modules = (nixos-2105.buildGoModule {
          inherit name src;
          vendorSha256 = "1idg6l9cnii6694kdmv9sc42c0bzk7pk67sdlglkaz52kg5hc6bd";
      }).go-modules;
  });
}

Error output:

``` these derivations will be built: /nix/store/d5ada6p0wl6pnz6xm4r5ws50yvha1c36-kopia-ede09c15854ce74fde40603a6857ed3991ba666c-go-modules.drv /nix/store/nvixyjbfji4ljlyqkhx32323ip5zq6a7-kopia-ede09c15854ce74fde40603a6857ed3991ba666c.drv building '/nix/store/d5ada6p0wl6pnz6xm4r5ws50yvha1c36-kopia-ede09c15854ce74fde40603a6857ed3991ba666c-go-modules.drv'... copying 1 paths... copying path '/nix/store/mbhc8w8qxckv4j1rwz7mlaljahv6m46w-kopia-ede09c15854ce74fde40603a6857ed3991ba666c-go-modules' from 'ssh://... building '/nix/store/nvixyjbfji4ljlyqkhx32323ip5zq6a7-kopia-ede09c15854ce74fde40603a6857ed3991ba666c.drv'... copying 1 paths... copying path '/nix/store/0y74vrvy799phqqwl781q9xadzqaj5ly-kopia-0.8.4-go-modules' to 'ssh://... unpacking sources unpacking source archive /nix/store/w9zd6h50l258cdzkmja19pfvzih51ywj-source source root is source patching sources configuring building Building subPackage ./. go: inconsistent vendoring in /build/source: cloud.google.com/go/storage@v1.15.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/Azure/azure-pipeline-go@v0.2.3: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/aws/aws-sdk-go@v1.38.41: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/dustinkirkland/golang-petname@v0.0.0-20191129215211-8e5a1ed0cff0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/fatih/color@v1.12.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/golang/protobuf@v1.5.2: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/klauspost/compress@v1.12.2: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/pierrec/lz4@v2.6.0+incompatible: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/pkg/profile@v1.6.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt github.com/sanity-io/litter@v1.3.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt gocloud.dev@v0.23.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt golang.org/x/crypto@v0.0.0-20210506145944-38f3c27a63bf: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt golang.org/x/net@v0.0.0-20210505214959-0714010a04ed: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt golang.org/x/oauth2@v0.0.0-20210427180440-81ed05c6b58c: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt golang.org/x/sys@v0.0.0-20210503173754-0981d6026fa6: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt google.golang.org/api@v0.46.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt google.golang.org/grpc@v1.37.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt cloud.google.com/go/storage@v1.14.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod github.com/Azure/go-autorest/autorest@v0.11.18: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod github.com/aws/aws-sdk-go@v1.38.13: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod github.com/fatih/color@v1.10.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod github.com/golang/protobuf@v1.5.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod github.com/google/wire@v0.5.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod github.com/klauspost/compress@v1.11.13: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod github.com/pkg/profile@v1.5.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod gocloud.dev@v0.22.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod golang.org/x/crypto@v0.0.0-20210317152858-513c2a44f670: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod golang.org/x/net@v0.0.0-20210316092652-d523dce5a7f4: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod golang.org/x/oauth2@v0.0.0-20210313182246-cd4f82c27b84: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod golang.org/x/sys@v0.0.0-20210320140829-1e4c9ba3b0c4: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod google.golang.org/api@v0.43.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod google.golang.org/grpc@v1.36.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod To ignore the vendor directory, use -mod=readonly or -mod=mod. To sync the vendor directory, run: go mod vendor ```

[nh2]

The approach from https://github.com/NixOS/nixpkgs/issues/86349#issuecomment-624489806 worked for me (replacing modSha256 by its successor vendorSha256):

{
  kopia_unstable =
    let
      version = "ede09c15854ce74fde40603a6857ed3991ba666c";
      src = nixos-2105.fetchFromGitHub {
        owner = "kopia";
        repo = "kopia";
        rev = version;
        sha256 = "0wdkwpsgmycbiq8dvng3d0ps5qp87b7k1xvw8rv2wk1qzrrp6d93";
      };
    in
      (nixos-2105.callPackage "${nixos-2105.path}/pkgs/tools/backup/kopia" {
        buildGoModule = args: nixos-2105.buildGoModule (args // {
          vendorSha256 = "1idg6l9cnii6694kdmv9sc42c0bzk7pk67sdlglkaz52kg5hc6bd";
          inherit src version;
        });
      });
}

[danielphan2003]

The approach from #86349 (comment) worked for me (replacing modSha256 by its successor vendorSha256):

{
  kopia_unstable =
    let
      version = "ede09c15854ce74fde40603a6857ed3991ba666c";
      src = nixos-2105.fetchFromGitHub {
        owner = "kopia";
        repo = "kopia";
        rev = version;
        sha256 = "0wdkwpsgmycbiq8dvng3d0ps5qp87b7k1xvw8rv2wk1qzrrp6d93";
      };
    in
      (nixos-2105.callPackage "${nixos-2105.path}/pkgs/tools/backup/kopia" {
        buildGoModule = args: nixos-2105.buildGoModule (args // {
          vendorSha256 = "1idg6l9cnii6694kdmv9sc42c0bzk7pk67sdlglkaz52kg5hc6bd";
          inherit src version;
        });
      });
}

What about using nixos-2105.kopia.override instead of nixos-2105.callPackage "${nixos-2105.path}/pkgs/tools/backup/kopia"? This works when Nix is in restricted mode (in my case I was using flake).

Also Go 1.17 is already out, and latest Tailscale requires it so it makes sense to use buildGoModule = args: (nixos-2105.buildGoModules.override { go = go_1_17; }) (args // { ... }); (waiting https://github.com/NixOS/nixpkgs/pull/127519 to merge, so we can ditch overriding Go in the first place)

[evanjs]

Original Comment

> > The approach from [#86349 (comment)](https://github.com/NixOS/nixpkgs/issues/86349#issuecomment-624489806) worked for me (replacing `modSha256` by its successor `vendorSha256`): > > ```nix > > { > > kopia_unstable = > > let > > version = "ede09c15854ce74fde40603a6857ed3991ba666c"; > > src = nixos-2105.fetchFromGitHub { > > owner = "kopia"; > > repo = "kopia"; > > rev = version; > > sha256 = "0wdkwpsgmycbiq8dvng3d0ps5qp87b7k1xvw8rv2wk1qzrrp6d93"; > > }; > > in > > (nixos-2105.callPackage "${nixos-2105.path}/pkgs/tools/backup/kopia" { > > buildGoModule = args: nixos-2105.buildGoModule (args // { > > vendorSha256 = "1idg6l9cnii6694kdmv9sc42c0bzk7pk67sdlglkaz52kg5hc6bd"; > > inherit src version; > > }); > > }); > > } > > ``` > > What about using `nixos-2105.kopia.override` instead of `nixos-2105.callPackage "${nixos-2105.path}/pkgs/tools/backup/kopia"`? This works when Nix is in restricted mode (in my case I was using flake). > > Also Go 1.17 is already out, and [`latest`](https://github.com/tailscale/tailscale/commit/ec2b7c7da63ca1d0b9c128d381fcf63b65471999) Tailscale requires it so it makes sense to use `buildGoModule = args: (nixos-2105.buildGoModules.override { go = go_1_17; }) (args // { ... });` (waiting #127519 to merge, so we can ditch overriding Go in the first place)

Stumbled across this comment after realizing I couldn't build tailscale, but before I realized that's what this comment was about (tailscale specifically)😂

Seems to work fine with override, and no wonder it was failing to build until I overrode the go version used.

Tailscale 1.16.0 Override

```nix tailscaleLatest = let version = "1.16.0"; src = pkgs.fetchFromGitHub { owner = "tailscale"; repo = "tailscale"; rev = "v${version}"; sha256 = "sha256-wrC6bvC1f6SFY/SwVUW1+xuF1sk2jA9apMpo4jE2bFw="; }; in (pkgs.tailscale.override rec { buildGoModule = args: pkgs.buildGoModule.override { go = pkgs.go_1_17; } (args // { inherit src version; vendorSha256 = "sha256-HJLT0CGUVzJZ56vS/v3FZ7svxyzZ+wlXvrC2MExTLM4="; }); }); ```

Thanks for the pointer on overriding go in buildGoModule, I had not dug into Tailscale changes that far and did not realize a new Go version was required 🤪

[ShamrockLee]

In #158486, I work around this issue by passing related attributes including veniorSha256 as inputs, so that they can be owerrided with the override attribute of the resulted package. An example can be found here:

https://github.com/ShamrockLee/nixpkgs/blob/singularity-apptainer/pkgs/applications/virtualization/singularity/generic.nix#L21-L24=

Nevertheless, I still hope that this issue can be fixed. If not, we should at least document the workaround.

Overlay-style overridable mkDerivation (#119942) might be one of the ultimate solutions to issues of this kind. See #171586.

[Lillecarl]

@Mic92 Do you have a complete example for those of us who are still learning the language? :)

    zabbix-overlay =
      (self: super: super.zabbix.agent2.overrideAttrs (
            let
              version = "5.4.11";
              src = super.fetchurl {
                url = "https://cdn.zabbix.com/zabbix/sources/oldstable/5.4/zabbix-5.4.11.tar.gz";
                sha256 = "sha256-mxEsUFlnDB8MHWX6zWZaETNzNW6ut9ro28hXGG490kQ=";
              };
            in
            rec {
              name = "zabbix.agent2";
              inherit src;
              go-modules = (self.buildGoModule {
                inherit name src;
                vendorSha256 = "sha256-4Z4Fgv9zmAwp3tEvHj8yLOWkFY/zFz5VfehSWCuIUcI=";
              }).go-modules;
            }
        )
      );

I'm getting an infinite loop here.

[Mic92]

@Mic92 Do you have a complete example for those of us who are still learning the language? :)

    zabbix-overlay =
      (self: super: super.zabbix.agent2.overrideAttrs (
            let
              version = "5.4.11";
              src = super.fetchurl {
                url = "https://cdn.zabbix.com/zabbix/sources/oldstable/5.4/zabbix-5.4.11.tar.gz";
                sha256 = "sha256-mxEsUFlnDB8MHWX6zWZaETNzNW6ut9ro28hXGG490kQ=";
              };
            in
            rec {
              name = "zabbix.agent2";
              inherit src;
              go-modules = (self.buildGoModule {
                inherit name src;
                vendorSha256 = "sha256-4Z4Fgv9zmAwp3tEvHj8yLOWkFY/zFz5VfehSWCuIUcI=";
              }).go-modules;
            }
        )
      );

I'm getting an infinite loop here.

Try packageOverrides instead to avoid infinite recursions due the overlay.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/inconsistent-vendoring-in-buildgomodule-when-overriding-source/9225/7

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/use-a-specific-version-of-terraform-in-a-nix-shell/27880/1