sum.golang.org: responding with a 410 fifteen minutes after publishing a new module version

[mvdan]

$ go version
go version devel +f7a80af935 Mon Nov 23 05:47:51 2020 +0000 linux/amd64
$ go get -d github.com/multiformats/go-multicodec@v0.2.0
github.com/multiformats/go-multicodec@v0.2.0: verifying module: github.com/multiformats/go-multicodec@v0.2.0: reading https://sum.golang.org/lookup/github.com/multiformats/go-multicodec@v0.2.0: 410 Gone
    server response: not found: github.com/multiformats/go-multicodec@v0.2.0: invalid version: unknown revision v0.2.0
$ GONOSUMDB='*' go get -d github.com/multiformats/go-multicodec@v0.2.0
$ GOPROXY=direct GONOSUMDB='*' go get -v -d github.com/multiformats/go-multicodec@v0.2.0

At the time of running these commands, I pushed the new version via git tag -a v0.2.0 && git push --tags over fifteen minutes ago. Why am I still getting a 410?

Note how skipping sum.golang.org alone, or skipping both the sumdb and proxy, make the go get work. These commands were run outside of any module.

At first I assumed I was holding something wrong, but I've triple-checked everything. You can see the version at https://github.com/multiformats/go-multicodec/blob/v0.2.0/go.mod. I realise that the tag says "five hours ago", because that's when I tagged it - but I only pushed the tag fifteen minutes ago.

[mvdan]

As an extra data point, https://pkg.go.dev/github.com/multiformats/go-multicodec@master renders properly and I can see a new enough version of master. The license is getting recognised, too.

cc @heschik @katiehockman

[mvdan]

I should also mention that this git repository is perhaps a bit special when compared to most other git repositories. The v0.2.0 tag does not share history with earlier tags like v0.1.6, as it was started from an entirely clean git history in the same repository. The older tags are still in the same git repository, just not in the same master branch.

It's possible that this confused sumdb. Perhaps it expects all tags to share commit history in some way.

[mvdan]

Someone at work points out that this is because I did a go get -d github.com/multiformats/go-multicodec@v0.2.0 before the tag had been picked up by the proxy, so sum.golang.org must have cached the "version does not exist" proxy answer for 30m without trying again for a while. This seems to be confirmed by how, all of a sudden, go get works and it's roughly been 35 minutes since the tag was pushed.

Regardless of whether this is expected behavior, it seems fairly counter-intuitive and frustrating to the end user. Perhaps the proxy and sumdb should be taught to invalidate such a cached state when a new version is published.

[mvdan]

To clarify, the timeline was:

1) Run go get module@version 2) Push the tag - I had forgotten to do this at the start, so step 1 errored as expected 3) Re-run go get module@version, which fails with the sumdb 410 4) Keep getting failures for 30m 5) go get module@version finally works a good while later

So it almost seems like I should git push --tags, wait a good 30-60s to ensure that the proxy will have picked up the tag, and only then start using go get. Because otherwise I might get sumdb stuck for thirty minutes.

[iand]

See also https://github.com/golang/go/issues/41986

410 seems a strong response to give for a tag that has never been seen by the sum server since clients are free to interpret that as a permanent status. 404 would more appropriate for unknown tags, leaving 410 for retracted ones.

[katiehockman]

For the caching, this is working as intended. Though I do find it a bit odd that the request worked for proxy.golang.org but not for sum.golang.org. https://github.com/golang/go/issues/34370#issuecomment-532784418 provides a bit of context about why we haven't lowered the negative cache time.

Perhaps the proxy and sumdb should be taught to invalidate such a cached state when a new version is published.

There is no way for the proxy or sumdb to know that a new version has been published without attempting to fetch it. Currently, the entire system works on a pull model, and there is no "push" model available for us to be informed of a new version.

from @iand :

404 would more appropriate for unknown tags, leaving 410 for retracted ones.

This is purely an implementation detail of proxy.golang.org and sum.golang.org, in order to work with out internal caching. 410 can be treated as a 404 in this context.

[katiehockman]

I'm going to close this as there is no further action to take at this time.

[mvdan]

Thanks for your quick response.

For the caching, this is working as intended. [...] I'm going to close this as there is no further action to take at this time.

I think closing this issue on the technicality that the cache isn't buggy is a mistake. The cache is an implementation detail that I shouldn't need to worry about, in general. The steps I shared above are pretty common, I think - that is, doing the first go get just a second too early, before the version is picked up by the proxy. And the experience is pretty frustrating - the services keep telling me that no such version exists, and all I can see on the website FAQ is:

Then explicitly request that version via go get module@version. After one minute for caches to expire, the go command will see that tagged version. If this doesn't work for you, please file an issue.

So, it's clearly been longer than a minute, hence I file the bug.

At the very least, if the confusing UX is here to stay, I certainly believe it should be clearly documented. Because, right now, the "after one minute" goes directly against my experience of seeing nothing for over half an hour.

[marten-seemann]

I've run into this problem so many times, so first of all, I'd like to thank @mvdan for raising this issue. It's been really annoying to have to wait for half an hour just to bubble up a version update, so my - not very sophisticated - workaround has been to wait for at least 5 minutes before go getting a new version I just released.

Keeping a stale cache entry for half an hour seems like a bad solution in general. Reducing that time to one minute wouldn't be a bad resolution of this issue. Even better would be reducing it to a few seconds with exponential backoff afterwards, but I'm not sure how much overhead that be in the cache implementation.

[katiehockman]

At the very least, if the confusing UX is here to stay, I certainly believe it should be clearly documented. Because, right now, the "after one minute" goes directly against my experience of seeing nothing for over half an hour.

That's fair, thanks for making this point about the docs. At a minimum, it may make sense for the documentation to be updated to more clearly state expectations. I'll re-open this so that we can determine the right course of action on this.

[ConradIrwin]

I just ran into this exact problem, and came here as instructed:

Then explicitly request that version via go get module@version. After one minute for caches to expire, the go command will see that tagged version.

A quick solution could be to change that FAQ answer to include the extra detail.

In order to improve our services' caching and serving latencies, new versions may not show up right away. After you have pushed a new semantically versioned tag to a repository you can explicitly request that version via go get module@version. After one minute for caches to expire, the go command will see that tagged version. If you accidentally request a version before a tag exists in the repository the absence of the version will be cached. Wait 30 minutes for caches to expire before trying to request that version again. If this doesn't work for you, please file an issue.

I'd still be a bit annoyed at myself for trying to fetch before pushing the tag, but at least it would save me a few minutes of retrying to see if it had been more than one minute yet.

[katiehockman]

Apologies for the delay in updating the documentation. I will do that now and it should be fixed soon. Thanks for everyone's patience.

[ConradIrwin]

Thanks! @katiehockman

[katiehockman]

Thanks everyone for your patience! The documentation at proxy.golang.org has been updated. Hopefully this can set more clear expectations, but let me know if it doesn't.

[mvdan]

Yep, it looks clear now, thanks! Personally I'd hope to see that 30-minute worst case scenario be a shorter delay, but I get that it's not an easy tradeoff :)

[paullaffitte]

One solution that could be easy to implement would be to allow users to ask to refresh the cache for a module when it's marked as unavailable. It's probably not perfect since it requires user interaction, but it seems to be a better solution than having to wait 30 minutes. Of course you can choose to pull without cache by disabling it with environment variables, but in the case you want several people to pull it, it looks cumbersome to have to tell all of them that they have to use a specific command/env.

Allowing user to invalidate the cache could fix some other issues, like when you push a tag and in the minute after you realize you didn't put the tag on the right commit (that can happens if you don't use a CI, and in early stage of a project, it can happens quite easily to not have automated the release process yet). Then the only solution is to create a new tag, and write somewhere "don't use this version, we didn't want to release it but now that it's done it's too late". Even if you delete the tag the version still exists and it can be quite annoying. Some may argue that "once a version is published, you should not update it", but I want to reply to that : if git allows to delete/move a tag, and go modules are based on git, who is go to tell git users how they should use git?

[matthewmueller]

I ran into this exact problem too:

• Forgot to push up tags
• Ran go get -u github.com/matthewmueller/some-package
• Pushed up the tags
• Re-ran go get -u github.com/matthewmueller/some-package@v0.12.9

It took me some trial and error to figure out what was going on. Sharing my usage report in case it's helpful. It's starts with the error. The error unfortunately doesn't guide you towards the answer:

go mod download: github.com/matthewmueller/some-package@v0.12.9: verifying go.mod: github.com/matthewmueller/some-package@v0.12.9/go.mod: reading https://sum.golang.org/lookup/github.com/matthewmueller/some-package@v0.12.9: 410 Gone
        server response: not found: github.com/matthewmueller/some-package@v0.12.9: invalid version: unknown revision v0.12.9

Googling around is difficult because the unknown revision error seems to happen most often on private modules. There it instructs you to mess with your git config and whatnot. I published a public package so these instructions didn't make sense.

Without much luck, I flailed around a bit:

• I changed my GOMODCACHE location since maybe it cached that response.
• I ran go mod download to see if that would be any different

Finally on a whim, I tried disabling go sum:

GONOSUMDB=* go get -u github.com/matthewmueller/some-package@v0.12.9

That did the trick. It now works as if nothing was ever wrong.

[codyoss]

I ran into this today as well when doing a go get something@latest. Maybe in the past I or someone had tried to pull the version that did not previously exist but I am unsure.

Edit:

I ran into this a second time today. This time while making a new submodule in my project, pushing tags, then updating a dependency and running go mod tidy. This time I assume this is because I tried to pull the dependency within a minute of publishing it, but I could be mistaken.

[marten-seemann]

Is there any update on this? It doesn't happen very often, but when it happens, having to wait for the cache entry to expire is such a productivity killer, especially if one is bubbling up changes through a chain of dependencies.

[XANi]

Still happens. Why the go tooling can't just skip the cache if it returns 404 ?

Honestly since introduction of sum.golang.org and proxying the experience of using the tools have been entirely for the worse.

Even trying to go around by requesting commit ID instead of tagged version doesn't, and still tries the inept cache:

-> ᛯ go get github.com/zerosvc/go-zerosvc@7324d5d                                                                                                          1
go: downloading github.com/zerosvc/go-zerosvc v0.1.6
go: github.com/zerosvc/go-zerosvc@7324d5d: github.com/zerosvc/go-zerosvc@v0.1.6: verifying module: github.com/zerosvc/go-zerosvc@v0.1.6: reading https://sum.golang.org/lookup/github.com/zerosvc/go-zerosvc@v0.1.6: 404 Not Found
    server response: not found: github.com/zerosvc/go-zerosvc@v0.1.6: invalid version: unknown revision v0.1.6

[janpfeifer]

Just bumped into this issue today (I think I can't find another explanation) for project github.com/janpfeifer/gonb.

For ~30 minutes I'm not able to build the docker of the distribution (it should accompany the release), because it can't fetch the new version.

[0xanonymeow]

I ran into this exact problem too:

• Forgot to push up tags
• Ran go get -u github.com/matthewmueller/some-package
• Pushed up the tags
• Re-ran go get -u github.com/matthewmueller/some-package@v0.12.9

It took me some trial and error to figure out what was going on. Sharing my usage report in case it's helpful. It's starts with the error. The error unfortunately doesn't guide you towards the answer:

go mod download: github.com/matthewmueller/some-package@v0.12.9: verifying go.mod: github.com/matthewmueller/some-package@v0.12.9/go.mod: reading https://sum.golang.org/lookup/github.com/matthewmueller/some-package@v0.12.9: 410 Gone
        server response: not found: github.com/matthewmueller/some-package@v0.12.9: invalid version: unknown revision v0.12.9

Googling around is difficult because the unknown revision error seems to happen most often on private modules. There it instructs you to mess with your git config and whatnot. I published a public package so these instructions didn't make sense.

Without much luck, I flailed around a bit:

• I changed my GOMODCACHE location since maybe it cached that response.
• I ran go mod download to see if that would be any different

Finally on a whim, I tried disabling go sum:

GONOSUMDB=* go get -u github.com/matthewmueller/some-package@v0.12.9

That did the trick. It now works as if nothing was ever wrong.

ty sir. this saved me! 🫡