Golang dependency fails to update from GitHub Enterprise GHE repo

wadey commented 1 week ago

Is there an existing issue for this?

[X] I have searched the existing issues

Package ecosystem

Go

Package manager version

No response

Language version

1.22

Manifest location and content before the Dependabot update

go.mod

The relevant section:

require (
    my-ghe.com/my-org/my-repo v1.3.1
)

dependabot.yml content

version: 2
updates:
  - package-ecosystem: "gomod"
    directory: "/"
    schedule:
      interval: "weekly"

Updated dependency

my-ghe.com/my-org/my-repo v1.3.1 -> v1.3.2

What you expected to see, versus what you actually saw

This is with GHE 3.12, but with an updated version of the dependabot runner. Both the dependency repo and the repo running dependabot are "public" in the GHE instance.

When checking for the updates for a Go dependency that is hosted on our GHE instance it appears to find the new version number (1.3.2 in the logs below), but then it appears to attempt usinattempt the request against the public github.com and get a 404 after trying the public go module proxy and then give up.

updater | 2024/06/26 14:28:50 INFO <job_18145> Checking if my-ghe.com/my-org/my-repo 1.3.1 needs updating
  proxy | 2024/06/26 14:28:50 [066] GET https://proxy.golang.org:443/my-ghe.com/my-org/my-repo/@v/list
  proxy | 2024/06/26 14:28:51 [066] 404 https://proxy.golang.org:443/my-ghe.com/my-org/my-repo/@v/list
  proxy | 2024/06/26 14:28:51 [068] GET https://my-ghe.com:443/my-org/my-repo?go-get=1
2024/06/26 14:28:51 [068] * authenticating git server request (host: my-ghe.com)
  proxy | 2024/06/26 14:28:51 [068] 200 https://my-ghe.com:443/my-org/my-repo?go-get=1
  proxy | 2024/06/26 14:28:51 [070] GET https://my-ghe.com:443/my-org/my-repo.git/info/refs?service=git-upload-pack
2024/06/26 14:28:51 [070] * authenticating git server request (host: my-ghe.com)
  proxy | 2024/06/26 14:28:51 [070] 200 https://my-ghe.com:443/my-org/my-repo.git/info/refs?service=git-upload-pack
  proxy | 2024/06/26 14:28:51 [072] POST https://my-ghe.com:443/my-org/my-repo.git/git-upload-pack
2024/06/26 14:28:51 [072] * authenticating git server request (host: my-ghe.com)
  proxy | 2024/06/26 14:28:51 [072] 200 https://my-ghe.com:443/my-org/my-repo.git/git-upload-pack
  proxy | 2024/06/26 14:28:51 [074] GET https://my-ghe.com:443/my-org/my-repo.git/info/refs?service=git-upload-pack
2024/06/26 14:28:51 [074] * authenticating git server request (host: my-ghe.com)
  proxy | 2024/06/26 14:28:51 [074] 200 https://my-ghe.com:443/my-org/my-repo.git/info/refs?service=git-upload-pack
  proxy | 2024/06/26 14:28:51 [076] POST https://my-ghe.com:443/my-org/my-repo.git/git-upload-pack
2024/06/26 14:28:51 [076] * authenticating git server request (host: my-ghe.com)
  proxy | 2024/06/26 14:28:51 [076] 200 https://my-ghe.com:443/my-org/my-repo.git/git-upload-pack
  proxy | 2024/06/26 14:28:52 [078] POST https://my-ghe.com:443/my-org/my-repo.git/git-upload-pack
2024/06/26 14:28:52 [078] * authenticating git server request (host: my-ghe.com)
  proxy | 2024/06/26 14:28:52 [078] 200 https://my-ghe.com:443/my-org/my-repo.git/git-upload-pack
  proxy | 2024/06/26 14:28:52 [080] GET https://proxy.golang.org:443/my-ghe.com/my-org/my-repo/@v/v1.3.2.mod
  proxy | 2024/06/26 14:28:53 [080] 404 https://proxy.golang.org:443/my-ghe.com/my-org/my-repo/@v/v1.3.2.mod
  proxy | 2024/06/26 14:28:53 [082] GET https://proxy.golang.org:443/sumdb/sum.golang.org/supported
  proxy | 2024/06/26 14:28:53 [082] 404 https://proxy.golang.org:443/sumdb/sum.golang.org/supported
  proxy | 2024/06/26 14:28:53 [084] GET https://sum.golang.org:443/lookup/my-ghe.com/my-org/my-repo@v1.3.2
  proxy | 2024/06/26 14:28:53 [084] 404 https://sum.golang.org:443/lookup/my-ghe.com/my-org/my-repo@v1.3.2
  proxy | 2024/06/26 14:28:53 [086] GET https://proxy.golang.org:443/github.com/my-org/my-repo/@v/list
  proxy | 2024/06/26 14:28:54 [086] 404 https://proxy.golang.org:443/github.com/my-org/my-repo/@v/list
  proxy | 2024/06/26 14:28:54 [088] GET https://github.com:443/my-org/my-repo/info/refs?service=git-upload-pack
2024/06/26 14:28:54 [088] * authenticating git server request (host: github.com)
  proxy | 2024/06/26 14:28:54 [088] 404 https://github.com:443/my-org/my-repo/info/refs?service=git-upload-pack
2024/06/26 14:28:54 [088] * auth'd git request returned 404, retrying without auth
  proxy | 2024/06/26 14:28:54 [088] * de-auth'd request returned 401, replacing response
  proxy | 2024/06/26 14:28:54 [090] GET https://github.com:443/my-org/my-repo
2024/06/26 14:28:54 [090] * authenticating git server request (host: github.com)
  proxy | 2024/06/26 14:28:54 [090] 404 https://github.com:443/my-org/my-repo
2024/06/26 14:28:54 [090] * auth'd git request returned 404, retrying without auth
  proxy | 2024/06/26 14:28:54 [090] * de-auth'd request returned 404, ignoring response

  ...

  updater | 2024/06/26 14:28:55 INFO Results:
Dependabot encountered '1' error(s) during execution, please check the logs for more details.
+-------------------------------------------------------------------+
|                   Dependencies failed to update                   |
+----------------------------------+--------------------------------+
| my-ghe.com/my-org/my-repo | git_dependencies_not_reachable |
+----------------------------------+--------------------------------+

versions:

Resolved ghcr.io/dependabot/dependabot-updater-gomod:v2.0.20240410143125@sha256:58c878b7beb77f957f6f8293d9da8edca9664bb964d9ef62ff75f5229af9d242 to existing ghcr.io/dependabot/dependabot-updater-gomod@sha256:58c878b7beb77f957f6f8293d9da8edca9664bb964d9ef62ff75f5229af9d242
  Resolved ghcr.io/github/dependabot-update-job-proxy/dependabot-update-job-proxy:v2.0.2024040[51](https://slack-github.com/slack/nebula-ca-client/actions/runs/484293/job/808673#step:5:52)34646@sha256:3c2ceb8ea54aa14b315b9f03fc38ef6cb2fa5cc94fdd83ca3dcddaf2563cae93 to existing ghcr.io/github/dependabot-update-job-proxy/dependabot-update-job-proxy@sha256:3c2ceb8ea54aa14b315b9f03fc38ef6cb2fa5cc94fdd83ca3dcddaf2563cae93

Native package manager behavior

No response

Images of the diff or a link to the PR, issue, or logs

No response

Smallest manifest that reproduces the issue

No response

wadey commented 1 week ago

Can I tell dependabot to set GOPRIVATE=my-ghe.com somewhere? I'm not sure if that is the cause of this issue. It doesn't look like that is possible though according to: #7254

Ideally for GHE the domain for the instance should be added to GOPRIVATE automatically I'd say.

But I still can't tell if this is the root issue here or not.

jakecoffman commented 1 week ago

Currently there's no way to set GOPRIVATE manually. The Dependabot service sets it based on a few factors:

If the update config has any registries it sets GOPRIVATE=*
If the repo is private it sets GOPRIVATE=${hostname}/${repo.account_name}/*

These assumptions aren't great so we'll have to work on that.

In the meantime I can offer you workarounds! You can set the repo to be private, or define any registry on the update config in the dependabot.yml:

version: 2

registries:
  workaround: # see https://github.com/dependabot/dependabot-core/issues/10086
    type: rubygems-server
    url: example.com
    token: fake-token
updates:
  - package-ecosystem: "gomod"
    directory: "/"
    schedule:
      interval: "weekly"
    registries:
      - workaround

Sorry for the inconvenience! 🙇

wadey commented 1 week ago

Thanks for the workaround! For GHE it would be great it if defaulted to GOPRIVATE=${hostname} I think. That should fix the issue for everyone using GHE.

wadey commented 1 week ago

ah that workaround as stated doesn't work, it complains with:

Tokens should be stored as a GitHub secret.

So it has to be more complicated with a fake secret added to the repo:

registries:
  workaround: # see https://github.com/dependabot/dependabot-core/issues/10086
    type: rubygems-server
    url: example.com
    token: ${{secrets.WORKAROUND}}

But I can confirm that the workaround solves the core issue, and dependabot creates the PR with the workaround in place.

dependabot / dependabot-core