Open mvdan opened 5 years ago
@mvdan @rogpeppe and I were discussing this offline including various alternatives to the GOPROXY=direct
"solution".
I think there is a change in workflow here that might affect a not-insignificant number of users. That's just a feeling, because I don't have specific numbers.
In 1.12, go get -u
works in the scenario described.
In 1.13, are we saying GOPROXY=direct go get -u
is only guaranteed way to deal with the outlined scenario?
I think there is a change in workflow here that might affect a not-insignificant number of users. That's just a feeling, because I don't have specific numbers.
Not that this adds much to the discussion, but I'd be one of those users. Thanks for raising this issue.
One option would be for the proxy not to serve branch names at all. It could serve a 404 or 410 for the .info
file for the branch, and then the default fallback to direct
would resolve that to the correct version.
We would lose some of the efficiency benefits of the proxy that way, but those benefits are most important for transitive dependencies — and transitive dependencies should always be pre-resolved to proper versions or pseudo-versions anyway.
CC @hyangah @katiehockman
One option would be for the proxy not to serve branch names at all.
This would presumably include @latest
so that also falls back to direct
?
@bcmills That is an interesting idea. Will the fallback to 'direct' mode be permanent? It seems like go commands are starting to depend on the fallback more and more. What do you think about tags which don't look like semver or incompatible? If we decide to take this path, maybe the go command even stops sending requests to proxy.golang.org for such requests to avoid extra network round trips.
@myitcv Is @latest
version meant the one from the go command's perspective (the default when version is not specified), or the proxy endpoint /@latest
for pseudo versions?
The go command currently picks up the latest version from the proxy's /@v/list
endpoints and if there is no released version, the go command hits /@latest
version to find any usable latest pseudo version.
If you meant the former, most go users will start fetching modules, packages, programs without version specified and the go command will treat them (and some of the dependencies) as @latest
. As a result, the users will still face the problems proxy wanted to solve. (e.g. left-pad issues, vcs client requirements, ...).
--
Someone suggested a GONOPROXY
trick for the repos I am actively working and testing on.
That will cause the module I am working on to be fetched directly (without any roundtrip to the proxy) but does verification against sum.golang.org
.
We are also planning to improve data freshness by utilizing the new 1.13 feature (#32239) to detect the latest pseudo version.
If we decide to take this path, maybe the go command even stops sending requests to proxy.golang.org for such requests to avoid extra network round trips.
I really like this idea. I'm slightly concerned about different versions of go
returning different versions for @latest
. The true latest is probably better than the proxy's latest, though.
@hyangah
... for pseudo versions?
I'm not sure this issue is specific to pseudo versions is it? Because it occurs whenever a VCS releases a new version that the proxy has not yet seen, pseudo-version or otherwise.
The issue (at least the aspect I'm worried about) is that in in 1.13, we are saying users either have to:
GOPROXY=direct go get -u
GONOPROXY
to avoid getting a stale latest version.
Both solutions work, but both represent a change to what is, I suspect, a common workflow.
I just want to be sure a) my understanding is correct and b) we are comfortable with pushing this change in workflow.
@myitcv For non-pseudo versions, proxy.golang.org includes it in the /@v/list
results as soon as it observes the new version. The module owners can teach proxy.golang.org about their new released version by simply running go mod download
or go get
with the newly released version. I hope that becomes part of the best practice for module release. The future go release
tool may be helpful.
Once the semver-tagged module lands in the proxy.golang.org, it should be available to other users almost immediately (a couple of minutes in worst case). Otherwise, please file a bug.
@hyangah
Will the fallback to 'direct' mode be permanent? It seems like go commands are starting to depend on the fallback more and more.
I'm not sure. I certainly expect it to be around for a while, at least.
What do you think about tags which don't look like semver or incompatible?
It seems fairly harmless to let those drop to direct
too, since the go
command will resolve them to canonical versions or pseudo-versions in the steady state.
If we decide to take this path, maybe the go command even stops sending requests to proxy.golang.org for such requests to avoid extra network round trips.
That would be fine for proxy.golang.org
, but not for proxies in general: for private proxies we still want to give the proxy the opportunity to reject the module outright (for security, privacy, or other policy reasons).
This would presumably include
@latest
so that also falls back todirect
?
Probably not, for the reasons @hyangah mentioned. Plus, @latest
is not as rare as, say, a branch-name or non-canonical tag: if someone runs go get -u
they'll end up fetching the latest
version of potentially a very large number of modules, so it's more important to get the efficiency boost from the proxy.
@bcmills my personal preference is also for proxy.golang.org not to accept /@v/*.info
queries with vcs branch names, non-semver tags, commit hashes or incompatible semver tags (whatever that is not resolved to the identical version). That is too vcs-centric. :-)
The module owners can teach proxy.golang.org about their new released version by simply running go mod download or go get with the newly released version
Ah, now I think I understand your reference to https://github.com/golang/go/issues/32239 (I couldn't understand why that field would be useful to the user, but it is to the proxy).
So just to clarify. Under your proposal:
go get $package@$version
against the proxy will, in the case of a "cache miss" on the proxy, cause the proxy to synchronously go get $package@$version
against the VCS before returning to the callerlatest
according to VCS, the proxy can then update its version of @latest
for that moduleIs that about right?
@myitcv Yes. right. For non-pseudo versions, proxy.golang.org "already" includes the newly found version in its results even without the fix for #32239.
Thanks for confirming @hyangah. This is, if memory serves, very much along the lines of a conversation we had before, so great to hear it's going live!
So, assuming we are post the release of a proxy fix that includes #32239, the change in workflow here is: when a new (pseudo) version is released, someone/something (i.e. go release
) needs to pull the version through the proxy.
That's definitely more understandable, but still worth flagging up in the release notes I suspect.
(On a related note: given that many (most?) people release new versions via one of the GitHub interfaces (e.g. "Draft new release") it might be worth kicking off a conversation with GitHub folks about integrating some sort of hook with the proxy?)
I hit this multiple times in my personal infrastructure. I usually push changes to a repository, and then rebuild a Docker container with go get
.
RUN go get github.com/FiloSottile/mostly-harmless/covfefe/cmd/webfefe@master
The proxy cache means that I can't deploy more than once every cache period.
My temporary solution is to set GONOPROXY
for the main module, which still gets me most of the performance advantage of the proxy.
# https://github.com/golang/go/issues/32870
ENV GONOPROXY github.com/FiloSottile/mostly-harmless
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
No, because Go 1.12 doesn't have a GOPROXY set up by default.
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
Module A and B live in separate repositories, and they are both v0. A depends on B.
I push a commit to B, and then run
go get -u
in A to update the dependency to that new master commit. Alternatively, triedgo get B@master
.What did you expect to see?
A's
go.mod
updated to show B's latest pseudo-version with the master commit just pushed.What did you see instead?
B's version staying at an older version, pushed hours or days before. Presumably because
proxy.golang.org
caches the version for@latest
and@master
.This went unnoticed for a while and someone was scratching their head until they realised
go get -u
hadn't done what they were used to.GOPROXY=direct go get -u
had to be used to work around this issue in the end. The other option was to manually copy the commit hash to force the proxy to fetch that git ref, which is a bit cumbersome.In the end, I worry that the new
GOPROXY
default in 1.13 is going to confuse people who are used to pushing to multiple repos that depend on each other. I understand why the proxy server can't fetch from git repeatedly every single time, but I wonder if there's something we can do to not silently break this./cc @bcmills @heschik @jayconrod