Proposal: improve UX for major module versions

[adg]

DEPRECATED

Please see the new proposals #40357 and #40323 that supersede this one.

---

Proposal: improve UX for major module versions

Peter Bourgon (@peterbourgon), Andrew Gerrand (@adg)

Problem statement

When a user wants to use a module, it is the v0/v1 version of that module which is most prominent, as it is selected by the base repository path: github.com/user/repo is in effect a constraint to v0.x.y/v1.x.y.

To use v2 or above, Semantic Import Versioning requires that the major version number is a suffix of the module path: github.com/user/repo/v2 (constrained to v2.x.y), github.com/user/repo/v3 (constrained to v3.x.y), and so on.

It’s easy for module consumers to default to v0/v1, even if that version is obsoleted by a more recent major version. Module consumers may even be totally unaware of later major versions.

Discoverability is a key issue. The mechanisms for module authors to advertise recent major versions are inconsistent, and can be low-visibility (documentation? README.md?) or highly disruptive (printing deprecation warnings in init, broken builds to force an investigation).

Abstract

We propose two improvements: one targeted at module consumers, and the other at producers.

For consumers, we propose a mechanism that notifies users of the latest major version of a module dependency when that dependency is first added to a project.

For producers, we propose adding a deprecated directive to go.mod files to signify the end-of-life of a major version.

These are just preliminary ideas which we hope to refine and improve in response to feedback gathered here.

Proposal 1: Notification of latest major version

We propose notifying users of new major versions when:

• They first add a requirement for an old major version of a module
• They update a requirement for an old major version of a module

There are a few ways users add requirements to their modules:

• By running go get github.com/user/repo
• By adding a require github.com/user/repo line to their go.mod file manually
• By adding an import line to a Go source file, and having goimports or gopls editor integration add the import line on save

For the latter two, the module isn't fetched until the go command is invoked within the module.

There are a few ways users update requirements:

• By running go get [-u] github.com/user/repo
• By running go get -u ./... from the module root
• Or to find out about available updates, they may run go list -m -u all

In each of these cases, the go tool plays a key role, and so we propose to make the go tool print a note if a requirement is being added when there is a more recent major version of the module available.

Examples

Consider a user fetching peterbourgon/ff with go get. We propose adding a notification to the output, alerting the user to a new major version:

$ go get github.com/peterbourgon/ff
go: finding github.com/pelletier/go-toml v1.6.0
go: finding gopkg.in/yaml.v2 v2.2.4
go: finding github.com/davecgh/go-spew v1.1.1
go: downloading github.com/peterbourgon/ff v1.7.0
go: extracting github.com/peterbourgon/ff v1.7.0
go: note: more recent major versions of github.com/peterbourgon/ff are available     👈
go: note: to install the most recent one, run `go get github.com/peterbourgon/ff/v3` 👈

Consider a user listing all of the most recent versions of their dependencies. We propose adding the latest major version alongside any new minor or patch versions:

$ go list -m -u all
example.com/my-module
github.com/BurntSushi/toml v0.3.1
github.com/mitchellh/go-wordwrap v1.0.0
github.com/peterbourgon/ff v1.6.0 [v1.7.0] <v3.0.1>                        👈 ONE OF
github.com/peterbourgon/ff v1.6.0 [v1.7.0] <github.com/peterbourgon/ff/v3> 👈 THESE
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7 [v0.0.0-20191204190536-9bdfabe68543]
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 [v1.0.0-20200227125254-8fa46927fb4f]
gopkg.in/yaml.v2 v2.2.2 [v2.2.8]

If a new requirement is added to a go.mod file manually, the go tool would print the notification when it first fetches the new module, as part of a go build, go test, etc. run.

Proposal 2: A new deprecated directive

Producer-side deprecation feature: add a deprecated line, which will cause the module to fail to fetch at that version, printing an error with the most recent non-deprecated version of the same major version.

module github.com/user/repo

deprecated

require (...)

If the go tool were asked to fetch this deprecated module, it would fail:

error: module github.com/user/repo@v1.7.1 is deprecated

If the Proposal 1 were adopted, the error message could be more useful:

error: module github.com/user/repo@v1.7.1 is deprecated
error: try the latest major version: go get github.com/user/repo/v2

The deprecated directive may optionally include a successor module identifier. If specified, the error printed when fetching this module version would also include a reference to the successor module. This can be used to point to the next major version:

deprecated github.com/user/repo/v2

Or to a new import path altogether:

deprecated gitlab.com/newuser/newrepo

If the go tool were asked to fetch this deprecated module, it would fail with a more explicit suggestion to the user:

error: module github.com/user/repo@v1.7.1 is deprecated
error: try its successor: go get github.com/user/repo/v2

If the successor module is also marked as deprecated and includes a successor module, the go tool might follow those links, and print the final non-deprecated module in the error message.

When package producers decide that a major version is deprecated, the intent is for them to create a new minor or patch release of that major version with the deprecated directive in the go.mod. This version will only be selected by the go tool when the corresponding major version is first added to a project, or if someone tries to update from an earlier version of that major version. In both cases, the go tool fails to fetch the module and provides a useful and actionable message: the user is instructed to pick a non-deprecated version.

Users can always use earlier versions of that major version, and MVS should ensure that the deprecated version will not be selected. If module A has a requirement for B at v1.7.0, and B v1.7.1 is later tagged as deprecated, A can continue to use B at v1.7.0, and the maintainer of A will only become aware of the deprecation when they try to update B.

It should not be possible for the go tool to mechanically add a deprecated version to a go.mod file.

A major version may be "un-deprecated" by publishing a subsequent minor or patch version without a deprecated directive in its go.mod. The specific deprecated version remains unusable, but the earlier and later versions still work normally.

Examples

A producer deprecating v1 of their module:

• Producer tags github.com/user/repo v1.0.0, v1.1.0, ... up to v1.7.0
• Producer tags github.com/user/repo/v2 v2.0.0, and soon afterward v2.0.1
• Producer adds a deprecated or deprecated github.com/user/repo/v2 line to the go.mod of the v1 branch and tags it v1.7.1

A consumer using a module that is then deprecated:

• Consumer has a module requirement require github.com/user/repo v1.7.0
• Producer tags the deprecated v1.7.1 version
• Consumer continues to use v1.7.0 without issue
• If the user tries to upgrade the dependency, the deprecation prevents it
• The user can keep their existing v1.7.0 or migrate to the new v2.x.y

A consumer tries to fetch a module at a deprecated major version:

• Consumer runs go get github.com/user/repo as a new dependency
• go get selects the highest version in the v0 or v1 trees, which is v1.7.1
• go get prints an error message:

error: module github.com/user/repo@v1.7.1 is deprecated
error: try the latest major version: go get github.com/user/repo/v2

• Consumer runs the suggested command, installing v2 of the module

Composition

One of Go’s strengths is the orthogonality of its features. We believe the proposed features compose nicely and strengthen each other.

Taken separately, the proposals can stand on their own: P1 provides package consumers with useful information without direct action from package producers; P2 allows package producers to give their consumers more specific guidance on an opt-in basis.

Taken together, the proposals enrich each other: P1 improves the error messages that accompany P2; P2 funnels users into the upgrade paths created by P1.

Integrations

pkg.go.dev

The Go package discovery website at pkg.go.dev shows modules and their versions. However, it obscures successive major versions when they exist, apparently treating major module versions as completely distinct. For example, the landing page for peterbourgon/ff shows v1.7.0 with a "Latest" bubble beside it. The versions tab does list other major versions, but under the heading "Other modules containing this package", which is confusing.

Instead, pkg.go.dev could feature a prominent indicator on the landing page for a v0/v1 module that there are two successive major versions (v2 and v3), to funnel the user toward the latter.

Editor integration (gopls, goimports)

Go text editor integrations typically include a feature that automatically adds import statements to source files based on the mentioned identifiers. Because of Semantic Import Versioning, this also gives those tools the responsibility of choosing the major version of the imported module. In the case where there is no suitable existing requirement in the project’s go.mod file, these editor integrations could alert the user to the availability of newer major module versions. How this works is outside the scope of this proposal.

Appendix

• Related proposals:
  • retract directive
  • go.dev: don't imply that vN.x.y is the latest version when vN+1 exists
  • go/list adds another flag to support showing major upgrade available for module

[mvdan]

Instead, pkg.go.dev could feature a prominent indicator on the landing page for a v0/v1 module that there are two successive major versions (v2 and v3), to funnel the user toward the latter.

I think this feedback is good, and it has been given before: https://github.com/golang/go/issues/37765

Assuming that pkg.go.dev will become more prominent, and eventually replace godoc.org, I wonder if this change would be enough to nudge users towards the latest major version of a module.

I get that it would not be as comprehensive as the other two changes proposed here, but it might be best to attempt fixing this UX problem in incremental steps, only considering the more aggressive/intrusive changes when the smaller steps prove to not be enough.

[adg]

@mvdan thanks for the pointer to that proposal; missed that in our review of related issues. Added to the appendix.

[kokes]

Regarding the second subproposal: What I'm wondering is whether the hard error that disallows installing of deprecated trees is future proof.

1. There is a tutorial on using library@v1. This library is at some point deprecated in favour of v2, but this is a breaking change, the API is different and the tutorial may no longer apply. When the user reads the tutorial and tries installing library@v1, it gives them "don't install this, it's deprecated, try v2 instead" - which might confuse them as v2 is not really usable due to the breaking change, but they cannot install v1 at all - unless they dig out the last non-deprecated v1.x.y version (from where? pkg.go.dev? do they know it can be done?). I don't have a clear solution in mind, but perhaps some override or more verbose errors might be helpful here.

2. Say I install v1.0.0 and it has a security issue, so v1.0.1 is released, fixing it (I haven't noticed, I'm still on v1.0.0). Even later on, v1.0.2 is released with a deprecated tag, because there's a v2 now. At this point, I have v1.0.0 locked in my go.(mod|sum), but there is no way for me to automatically update to a safe version with the same API (I can't use v2).

Both of these cases could use a "give me the last non-deprecated version within this major version" - because erring out does not help (in installing/in fixing a security issue) and suggestion to use a new major version doesn't either, because of the potential API changes.

Also this is sort of breaking the semver contract of "patch versions don't mess things up" - since the newest patch version cannot be installed at all - this would break tools that automate dependency security updates (think dependabot, snyk) - when a new patch version gets released (with just a deprecation tag), the tool that tries to suggest an update fails in a weird way. Maybe I'm misunderstanding what happens when go get -u pkg gets run.

I hope I understood the flows correctly. In any case, thanks for this proposal, I love UX improvements.

[adg]

@kokes one thing we discussed, but didn't include in this proposal, was including in the deprecation error message a mention of the previous minor/patch version in addition to the pointer to the next available major version. Something like:

error: module github.com/user/repo@v1.7.1 is deprecated
error: try the latest major version: go get github.com/user/repo/v2
error: or the previous patch version: go get github.com/user/repo@v1.7.0

(This is just off the top of my head - such a message should be finessed to emphasise the major version, while still offering a path to the previous working version.) I think that would address the cases you describe.

With regard to external tools, they'd need to understand deprecation as much as it affects their operation, the same way they need to understand various other aspects of Go modules.

[Merovius]

To suggest an alternative (haven't thought it fully through, though - you might have considered this and rejected it for reasons I can't think of right now):

• If a tool initially is asked to add example.com/foo (without any version-signifier) to go.mod, it automatically adds the latest major version instead (e.g. go get example.com/foo might add require "example.com/foo/v3" to go.mod).
• If an explicit version-signifier is given, the latest version in that major line is used, even if it is not the latest (e.g. go get example.com/foo/v2 would add require "example.com/foo/v2 to go.mod, even though v3 also exists).
• Even though v0/v1 as version-signifiers are normally omitted, they can also be used for this purpose.

I think this would solve the same issues, in that users doing the obvious thing without checking which versions exist would still default to the latest. But in this more common case no extra step of mentally parsing the output and re-entering/changing is needed. The less common case (for some reason I specifically want to use an older version) is still allowed.

It should not be possible for the go tool to mechanically add a deprecated version to a go.mod file.

I'm not super against this, but I would prefer if a satisfying solution can be found that doesn't include this. I would just really like to avoid having to manually touch go.mod if at all possible. And for better or for worse, there can be reasons to specifically use a deprecated version, even in a new project. For example, major versions might be using different wire- or disk-encodings and I'm bound by existing services not having migrated yet. That reason IMO also prohibits any hard failures when using/upgrading to deprecated versions in general: I might want to deprecate v1 (to make clear new users shouldn't use it) but still maintain it for users that do need it.

[peterbourgon]

@Merovius

If a tool initially is asked to add example.com/foo (without any version-signifier) to go.mod, it automatically adds the latest major version instead

SIV dictates that example.com/foo is semantically equivalent to the larger of extant example.com/foo/v{01}. This might be fudge-able, but folks still have unqualified import statements in their code that must continue to work. I think for these reasons this is a non-starter. (Maybe other reasons, too.)

I would just really like to avoid having to manually touch go.mod if at all possible.

I don't think any use case described in these proposals requires manually editing go.mod?

And for better or for worse, there can be reasons to specifically use a deprecated version, even in a new project.

Deprecation would apply only to specific, fully-qualified versions. The behavior described by the proposal as "deprecating an entire major version tree" is emergent from the go tool's selection of the highest qualifying specific version of a major version tree upon first import by a consumer.

Concretely, if you have (say) v1.7.1 which is not deprecated, you would need to tag (say) v1.7.2 as deprecated, you couldn't retroactively apply the deprecated bit to v1.7.1. Consumers currently using v1.7.1 would continue to function without impact; new consumers could explicitly request v1.7.1 if they really needed to use the v1 major version tree.

Which is all to say: it's still perfectly possible to use a previous specific version of a module, even if the most recent version in the major version tree has been marked as deprecated. (It's also possible to un-deprecate a major version tree, by tagging a new specific version with the deprecated directive removed from the go.mod.)

[earthboundkid]

If a tool initially is asked to add example.com/foo (without any version-signifier) to go.mod, it automatically adds the latest major version instead

I often work by just typing an import path into a file and then later running go mod tidy. In that scenario, I don't know when there would be a good time for the tool to move me automatically to v2+, as opposed to just suggesting it when I run go mod tidy or as feedback to gopls. If I actually did want to stay on v1, I would not like the tool to move automatically to a newer version behind my back.

[Merovius]

@peterbourgon

SIV dictates that example.com/foo is semantically equivalent to the larger of extant example.com/foo/v{01}. This might be fudge-able, but folks still have unqualified import statements in their code that must continue to work.

Not to nitpick, but by name, it dictates that for imports. But I'm not talking about imports, I'm talking about arguments passed to go get as "I would like to use this thing". That's at least my workflow, when I want to include a new dependency: I use go get to download the module and add it to go.mod, then I add an import via goimports when using packages. I'm not suggesting that an import-statement of example.com/foo could trigger the addition of example.com/foo/v2 to go.mod.

I agree that the nomenclature isn't entirely precise and might very well contradict the possibility of this, if taken literally. Personally, I don't really care about nomenclature though, but mostly whether the semantics make sense :) And I don't see where what I imagine breaks existing users.

To clarify: Say, I'd write a tool called "add-go-module", which, when given the name of a module (with or without version-specifier respectively) behaves as I described to insert a require directive into go.mod, would you consider that tool broken (or breaking code)? If yes, how? If not, what specifically would break if that tool is called go get instead?

I don't think any use case described in these proposals requires manually editing go.mod?

I don't understand this. The proposal seems to clearly state that the go tool should be prohibited from mechanically adding a deprecated version. So ISTM that if I want to add a deprecated version, I would have to add that manually.

Consumers currently using v1.7.1 would continue to function without impact; new consumers could explicitly request v1.7.1 if they really needed to use the v1 major version tree.

But not v1.7.2, correct? Otherwise I don't understand this sentence:

If the go tool were asked to fetch this deprecated module, it would fail with a more explicit suggestion to the user:

This seems to strongly imply that (part of?) the build fails if I want to use a deprecated version of a module? At least the first time I add it?

The use-case I was talking about is specifically to maintain a v1 branch for users who have not yet been able to migrate to v2 - including realeasing bugfixes and the like on that branch (say, as v1.7.3 etc), while still deprecating v1, so new users don't use it unconsciously. I understand if that use-case is not something we might want to jump through extra hoops to make possible. But I do feel we shouldn't add new code to specifically make it impossible, unless there's a good reason.

[peterbourgon]

@Merovius

The proposal seems to clearly state that the go tool should be prohibited from mechanically adding a deprecated version. So ISTM that if I want to add a deprecated version, I would have to add that manually.

Consumers should never be allowed to add explicitly deprecated versions. The go tool should refuse to add them to the go.mod, and if they are manually placed in the go.mod, the go tool should refuse to compile them.

The use-case I was talking about is specifically to maintain a v1 branch for users who have not yet been able to migrate to v2 - including realeasing bugfixes and the like on that branch (say, as v1.7.3 etc), while still deprecating v1, so new users don't use it unconsciously.

Ah. This use case is not a match for the semantics of deprecated as we have laid them out. According to this proposal, if you are actively maintaining a v1 branch with bugfixes, it is necessarily not deprecated.

[sylr]

The use-case I was talking about is specifically to maintain a v1 branch for users who have not yet been able to migrate to v2 - including realeasing bugfixes and the like on that branch (say, as v1.7.3 etc), while still deprecating v1, so new users don't use it unconsciously.

You want to maintain a deprecated branch ? that seems paradoxical but if you really want to then:

v1.7.0 (not deprecated) -> v1.7.1 (deprecated) -> v1.7.2 (patch release not deprecated) -> v1.7.3 (deprecated).

each time you patch your deprecated branch you release a tag which is not deprecated then you release a deprecated one just after.

[fatih]

When package producers decide that a major version is deprecated, the intent is for them to create a new minor or patch release of that major version with the deprecated directive in the go.mod.

Is this intent a SHOULD or MUST? Can a producer deprecate a package without bumping the patch or minor version?

[peterbourgon]

@fatih

Can a producer deprecate a package without bumping the patch or minor version?

No, deprecation is defined as the go.mod including the deprecated directive, and that change, like any other semantic change to a module, can't be applied to existing versions, only new ones.

[Merovius]

This use case is not a match for the semantics of deprecated as we have laid them out.

That's why I mentioned that I don't like those semantics.

You want to maintain a deprecated branch ? that seems paradoxical that seems paradoxical

I disagree. The definition of "deprecated" I am using is consistent with what Google gives:

(chiefly of a software feature) be usable but regarded as obsolete and best avoided, typically because it has been superseded.

I don't think "regarded as obsolete and best avoided" implies "can't be used". In the end, what "best avoided" means is a tradeoff. If it where an actual absolute, you already have that semantic: Remove the public API - builds will break and no one will use it. Obviously that is too strict an interpretation. I don't see why "[edit]don't[/edit] use it if you can avoid it, but I will still offer some support if you can't" is so weak an interpretation as to be "paradoxical".

each time you patch your deprecated branch you release a tag which is not deprecated then you release a deprecated one just after.

To me, the patch release is still deprecated though. But I understand that it's technologically possible to work around the restriction imposed by the definition of deprecation in the proposal.

[sylr]

I don't think "regarded as obsolete and best avoided" implies "can't be used".

Only tags having the deprecated directive wouldn't be usable. That way it ensures that, if the latest tag of a branch is deprecated, people are aware the whole branch is obsolete (because of the error go would throw upon getting by default the latest tag of the branch).

If they still want to use that branch then they should go get explicitly last (or any for that matter) not deprecated tag of the branch.

[zachgersh]

I don't yet have a set of specific comments on this particular proposal (though I am very happy this is being discussed).

These proposals could be implemented separately and it seems like maybe they should be two separate issues (rolling this much into one proposal makes it tough for people to focus on their comments)?

Proposal 1 encompasses what I've suggested here: https://github.com/golang/go/issues/38502 (I'd be happy to expand it to incorporate go get) and take any suggestions from @adg @peterbourgon (I think we all want the same things).

Maybe we slim this issue down to just Proposal 2?

[sylr]

FYI I've gone from go 1.14 to 1.11 and I managed to build a project that imports a deprecated tag and it built normally.

$ mkdir -p tmp/mybin
$ cd tmp/mybin
$ go mod init mybin
$ go get github.com/sylr/go-mod-deprecated@v1.0.6
$ cat <<EOF > main.go
package main

import (
    deprecated "github.com/sylr/go-mod-deprecated"
)

func main() {
    if deprecated.IsDeprecated() {
        println("Deprecated")
    }
}
EOF
$ for go in go1.{11..14}; do $go run .; done
Deprecated
Deprecated
Deprecated
Deprecated

This proves that all modules aware go releases would be compatible with this new directive.

[wagslane]

I love the proposal to add error warnings when importing a module that isn't the latest major version.

However, I'm skeptical that that should be the default behavior, I would prefer to get the latest major version by default. If someone knows of a link as to why the latest isn't the default I would love to read it.

The second proposal grinds my gears a bit. If I understand correctly - the proposal would make the tool error out when trying to import a package that has been marked as deprecated? Seems problematic. Sometimes users have good reason to ignore deprecations (This isn't the JS community, we don't turn off bitwise operators because usually the dev meant to use the logical operator).

I would be in support of the deprecated directive if it resulted in a simple warning as well, rather than an error.

[jimmyfrasche]

I had to read proposal 2 a few times. It makes sense if the only change is to go.mod when you add the deprecation directive, but that wasn't immediately clear to me.

Deprecation isn't a good name for the proposed semantics. This is for the step after where the earlier versions are no longer maintained at all. I'd expect a deprecation to be a warning that what's described here is on the horizon.

[peterbourgon]

@lane-c-wagner

I would prefer to get the latest major version by default. If someone knows of a link as to why the latest isn't the default I would love to read it.

Semantic Import Versioning as implemented in Go modules creates ambiguities when a user types an unqualified import path e.g. github.com/user/repo. Theoretically, and in import statements, this necessarily means: take the most recent available of github.com/user/repo/{v0,v1} — tooling does not have the freedom to interpret it in any other way. When provided to the go tool in an e.g. get statement, that requirement is not necessarily strict, but it is my belief that defaulting to the most recent major version would be confusing, and the negatives of that confusion outweigh the positives of the default assumption — especially when an actionable notification has much lower cost, and almost as much benefit.

Sometimes users have good reason to ignore deprecations . . . I would be in support of the deprecated directive if it resulted in a simple warning as well, rather than an error.

It is important that the second proposal does not merely warn users that a module version is deprecated, but actively prevent them from using it. The goal of the second proposal is to place extra power into the hands of package producers, to allow them to express constraints on their consumers that are not currently possible. Any lesser effect on consumers reduces the value of the proposal to near zero.

But, this doesn't mean that package consumers can't effectively ignore the deprecation warning. They simply have to declare their dependency on a previous, un-deprecated version in the same major version tree of the module. The immediately previous version will have equivalent functionality, minus the deprecated bit.

The thesis of these proposals: if a consumer adds github.com/user/module to their project without understanding the SIV/v0/v1 implications of that identifier (i.e. effectively everybody) and a more recent major version of that module exists, then they should always be warned that they're on an old version (Proposal 1) — and, if the package producer has opted-in to it, even prevented from using that version unless they put in special effort (Proposal 2). —

@jimmyfrasche

Deprecation isn't a good name for the proposed semantics. This is for the step after where the earlier versions are no longer maintained at all. I'd expect a deprecation to be a warning that what's described here is on the horizon.

I think we're open to using a different name than deprecated, if that's the sticking point — spitballing, maybe discontinued? — but the thing you describe is already perfectly possible via normal means of documentation. An English-language deprecation warning in the package docs and/or the README, accomplishes what you want with essentially equivalent force of impact.

[jimmyfrasche]

Discontinued is a much better name for what's described here. I think my major point of confusion was why a deprecated version would be an error—but a discontinued version being an error makes perfect sense.

[bep]

I think this is a good proposal, and I may stretching it a little when I say that the "UX major module version" thing goes beyond the go command.

Currently, going from v1 to v2 would, in GitHub terms, mean either to create a v2 branch or a v2 folder -- neither would, by default, make the v2 (the latest) the default branch you see/clone.

The above is a major concern -- and you could probably back it by stats telling that Go is a language where modules rarely get to version 2 and above.

I would suggest something in the line of: A major version number must always be present in the import path, else you get the latest major version -- which will leave v2 versions as simple tags.

[adg]

A major version number must always be present in the import path, else you get the latest major version

@bep that's kind of a non-starter because v0/v1 do not have the major version in the path, and the go tool should do what you ask it to, not what it thinks you mean.

[Merovius]

but it is my belief that defaulting to the most recent major version would be confusing, and the negatives of that confusion outweigh the positives of the default assumption — especially when an actionable notification has much lower cost, and almost as much benefit.

FWIW, my first comment comes exactly from my disagreement with this. The title of this issue is "improve UX for major version modules". And IMO, the setup as suggested in the proposal is just not a good UX at all. I would argue that in >>90% of cases, a user wants the newest major version. And downloading the wrong version and showing me a message "please retype with this changed version number" is going to frustrate me. What's more, I think that most of the time releasing a v2 will also mean deprecating v1 - which is supposed to not even work, with the second proposal. So the default behavior is one that will most of the time be specifically broken. Lastly, there is nothing more frustrating to me than software telling me it can't do the thing I'm telling it to when there's no technical reason why it can't. Which is what's happening with deprecated modules failing. Overall, the proposal just feels like it will produce many moments of frustration for me in the future (like, for example, the fact that I can't copy-paste a url to go-get, because of the https - small moments of frustration, even if they have a logical explanation).

You say you'd find it confusing if the newest major version was used and that's subjective and thus fair enough. I just don't understand the confusion, because to me it seems the most natural and obvious thing to happen. And to me, the default should reflect the overwhelming majority of use-cases.

I would even be happier if instead of printing a warning, the go tool would ask you interactively what version you want. I don't think that's a practical suggestion (it's used non-interactively far too much for that), but it would be a significantly better UX than what's proposed.

It is important that the second proposal does not merely warn users that a module version is deprecated, but actively prevent them from using it.

Here's a question: Why not commit an empty .go file as the deprecated version then? It won't build and won't be usable. No change in tooling necessary. The error message will probably be bad. But the effect will be the same (FTR "because we have to maintain compatibility according to the rules" is not an answer - whether v1.7.2 breaks because it was called "deprecated" or because its APIs aren't there is immaterial to the breakage, as far as the rules are concerned).

The goal of the second proposal is to place extra power into the hands of package producers, to allow them to express constraints on their consumers that are not currently possible. Any lesser effect on consumers reduces the value of the proposal to near zero.

I think that's correct. To be explicit: The current best alternative (apart from above "tagging an empty package") is to mark all exported APIs as deprecated. The value a warning for deprecated packages would still provide is mainly, that the user doesn't have to explicitly call go vet to notice the deprecation, but that anyone fetching a recursive dependency would see the warning (so it would be far louder). It's a benefit, it's not zero, but I agree that it's relatively small. So I can see why you don't see it as worthwhile to weaken the proposal here.

Personally, the corollary to me is, that Proposal 2 just shouldn't happen. I find the current form not acceptable and if a weaker form isn't worth pursuing, then I wouldn't like either. Of course that's just my opinion.

Currently, going from v1 to v2 would, in GitHub terms, mean either to create a v2 branch or a v2 folder -- neither would, by default [sic], make the v2 (the latest) the default branch you see/clone.

Just to state this clearly: As a module publisher, this is fixable by setting the current branch in github (or whatever hosting you use) accordingly. Yes, it's something module publishers need to be aware of, but it's not something that any user or even developer needs to know. It's a one-time cost to pay.

[bep]

Just to state this clearly: As a module publisher, this is fixable by setting the current branch in github (or whatever hosting you use) accordingly.

That is a statement with a whole lot of assumptions. The master branch has its own set of meaning (in scripts etc.) -- changing the default on GitHub does not propagate to the rest of the world (including my muscle memory).

[elioengcomp]

A solution for this also needs to be proxy friendly. There should be a way to get deprecated modules using regular Go tooling so Go Modules proxies can fetch and serve those versions as well. It is not clear to me in the proposal if commands like go mod download would fail when executed against deprecated modules, but if that is the case we could have a flag to instruct that command to ignore module deprecation statements.

[neild]

Producer-side deprecation feature: add a deprecated line, which will cause the module to fail to fetch at that version, printing an error with the most recent non-deprecated version of the same major version.

There is an existing mechanism for marking a package deprecated: Add a "// Deprecated: " package doc comment. For a module containing multiple packages, do this for each package.

Deprecation comments are advisory. Causing the module fetch to fail is substantially more severe; it's equivalent to checking in a new version of the package with a just an empty .go file, modulo a better error message. This is a profound shift in the meaning of "deprecated"--instead of "don't use this, but it still works", it is now "deliberately broken".

What happens if a deprecated module needs a security fix? Presumably, you need to release a new, non-deprecated version followed by a re-deprecation. Will users discover that fix, especially after they have been trained to avoid "go get -u" for this module?

Philosophically, I am dubious about the release policy that module deprecations seem to encourage. The Go standard library has gone for ten years without breaking changes. When we released a new version of the protobuf module, we went to great lengths to preserve the old API as a wrapper of the new one to avoid imposing unnecessary toil on existing users. This is the standard we should strive for; new major versions come at a tremendous cost to users, should be considered only as a last resort, and should provide a path for users of the previous version to upgrade at their own pace or not at all. Providing tools to make it easier to write off old users strikes me as a step in the wrong direction.

[peterbourgon]

@neild

The Go standard library has gone for ten years without breaking changes. When we released a new version of the protobuf module, we went to great lengths to preserve the old API as a wrapper of the new one to avoid imposing unnecessary toil on existing users. This is the standard we should strive for; new major versions come at a tremendous cost to users, should be considered only as a last resort, and should provide a path for users of the previous version to upgrade at their own pace or not at all.

No. Emphatically no.

Some modules are produced by teams of engineers, used widely, and the cost of API breakage is high. These modules are important and a package management system needs to accommodate them, but they represent a superminority of modules by count. The behaviors you describe as appropriate for those modules are not universal.

The majority of modules produced do not have teams of people working on them. They aren't widely consumed by a significant proportion of all projects. The cost of API breakage is not high. Most, even overwhelmingly most, Go modules fit this description. Tagging a release of one of these modules doesn't and shouldn't represent a permanent and inviolable contract. Tagging a release is marking a point in a module's speculative and living evolution, which the producer offers to the community in hopes that it's useful, and, critically, that the producer needs to be able to rescind when they realize errors in their design. Producers need this ability because they are rarely a team of engineers with extraordinary resources: they are, overwhelmingly, single individuals, working with limited resources that they need to carefully allocate. They can't and shouldn't be expected to maintain all major releases of their modules into perpetuity.

To emphasize: the overwhelming number of modules produced and consumed in the Go ecosystem are not like the stdlib, or the protobufs library, or the AWS SDK. They are in many ways entirely opposite. They are small, narrowly focused, and always in some sense experimental. Being able to deprecate (or, if you prefer, discontinue) versions of these modules as the producers and consumers iterate and improve on design is necessary. (In fact, breaking changes in modules "at the edges" of a software ecosystem aren't in any way bad: they're necessary, actually good, almost a perfect proxy for the healthiness of that ecosystem.) It is not only inappropriate but actively harmful to burden all modules with API compatibility promises — in your terms, a Philosophy — appropriate only for the largest and most widely consumed software.

Critically, Proposal 2 does nothing to get in the way of modules for which stricter API compatibility guarantees are appropriate. It also doesn't get in the way of consumers using a discontinued lineage of a module. It only provides additional, important, and until now missing, levers for the vast majority of module authors for whom strict API compatibility is more harmful than helpful.

[daenney]

I don't particularly like the workflow that follows from the deprecation strategy suggested here.

Suppose I have a v1.7.0. I add the deprecated directive to go.mod, tag v1.7.1 and publish. Two years later a serious enough issue shows up that I decide that even though v1 has been deprecate for a long time, it warrants fixing. I now have to:

• Remove deprecate from go.mod
• Do my fix
• Tag v1.7.2
• Add deprecated back to go.mod
• Tag v1.7.3

To me that's a lot of bureaucracy to go through just to get a fix out. I can also easily see myself screwing up any of these steps. I could end up forgetting to remove or add the deprecated directive back at any point.

It's also weird to me to be de-deprecating and re-deprecating something in a cycle like this. At no point did v1 become supported again from my point of view, but having v1.7.2 without the deprecated directive does suggest that development of v1 has picked up again. I'm just trying to be nice and provide a fix in case you're still stuck on v1 for whatever reason, but this flow sends mixed signals.

Aside from the fact that I find the whole flow peculiar, I'm wondering if we can get some clarification on the following: if I am on v1.7.0, and go a go get -u ..., what happens? Should it update to v1.7.2 b/c that's the last non-deprecated version within the same major, or will it tell me I'm being a bad person and I should upgrade instead to a new major? At the very least I'd want the tooling to upgrade users to v1.7.2, to ensure they get that fix, but based on my understanding right now the tooling would pick up on v1.7.3, see it's deprecated and tell people to upgrade to a new major instead?

The other concern I'd like to raise is that it's very possible tooling, like Go module proxies, will try to optimise things by going down the path of "once I've seen a deprecated version, no new versions for this package will be released". Arguably that's an implementation bug, but I don't see an easy way to prevent it and potentially cause weird issues for people depending on what proxy they're using.

[peterbourgon]

@daenney

if I am on v1.7.0, and go a go get -u ..., what happens?

The go tool will try to upgrade to the latest version of the v1 major version tree, v1.7.3, which is marked as deprecated. The automated update will fail, and print an error message instructing you to explicitly request either the latest non-deprecated version on the v1 major version tree (which will be v1.7.2, including the security fix) or, ideally, upgrade to the most recent major version tree (maybe that's v3.0.2 or something) which is actually being maintained.

Deprecation (or discontinuation) is meant to be a way of telling consumers that no further work will be paid to this major version tree. In my mind this includes security updates. If your module is such that the concept of security updates applied to outdated major version trees is likely, then you should not use the deprecated (discontinued) feature. If you judge it unlikely enough that you apply the deprecated (discontinued) feature to a major version tree but then later find a vulnerability worth fixing, this bit of ceremony doesn't seem arduous.

it's very possible tooling, like Go module proxies, will try to optimise things by going down the path of "once I've seen a deprecated version, no new versions for this package will be released"

This would not be a valid assumption.

[daenney]

If your module is such that the concept of security updates applied to outdated major version trees is likely, then you should not use the deprecated (discontinued) feature.

This feels like a weird thing to leave out. I'd still want new consumers to start at new versions, and existing consumers to be notified that there's newer versions available and nudged to upgrade. How do I then go about signaling this to users?

[sylr]

I think people miss that:

• As package producers, they don't have to use it if they don't like what it implies.
• As package users, they can always bypass the depreciation once they received the warning (unless someone released patches under a still deprecated tag).

[peterbourgon]

I'd still want new consumers to start at new versions, and existing consumers to be notified that there's newer versions available and nudged to upgrade. How do I then go about signaling this to users?

Proposal 1 (notification) would provide these signals to consumers automatically, with no action on your part. Proposal 2 (deprecation/discontinuation) gives you something stronger than a notification, if you opt-in to using it.

[peterbourgon]

@elioengcomp

There should be a way to get deprecated modules using regular Go tooling so Go Modules proxies can fetch and serve those versions as well.

I don't quite understand why. Module proxies may need to fetch deprecated/discontinued modules in order to learn about them, but they should never serve them, because a deprecated/discontinued module should fail to be selected by MVS, and fail to compile.

[Merovius]

The cost of API breakage is not high. Most, even overwhelmingly most, Go modules fit this description.

As someone who has, as a pure user, suffered many times from API breakages in dependencies of software I use, I find this very hard to imagine.

They cannot and should not be expected to maintain all major releases of their modules into perpetuity.

I do not understand how "I can't deliberately break this version" can possibly be translated to "I promise to maintain it in perpetuity".

All of the things you mentioned are (to me) fully agreeable reasons to introduce API breakages during development or to stop maintaining old versions. None of them is even close to a reason for deliberately breaking them.

I think people miss that:

• As package producers, they don't have to use it if they don't like what it implies.
• As package users, they can always bypass the depreciation once they received the warning (unless someone released patches under a still deprecated tag).

In the same vein, without these proposals, package producers can already commit an empty repository and tag it, instead of using the proposed "deprecated" tag for exactly the same effect. It's not as good. But neither is having to re-type commands because the go tool refuses perfectly reasonable requests. I think instead of telling us we can work around the worse tooling, trying to find a solution with more common ground is advisable.

[peterbourgon]

All of the things you mentioned are (to me) fully agreeable reasons to introduce API breakages during development or to stop maintaining old versions. None of them is even close to a reason for deliberately breaking them.

Proposal 2 provides module authors a stronger way of asserting that an old version is no longer maintained than purely advisory. It is opt-in, and does not break consumers.

[Merovius]

It is opt-in, and does not break consumers.

It does break consumers. It is (still) functionally equivalent to committing an empty .go file. Either no change breaks consumers ("they can always just stay on a non-broken version, if they want to") or this change breaks consumers (updating to a version tagged as deprecated will break the build, just like updating to a version that removed all public identifiers).

I do not understand why you continue to ignore what is being said - by multiple people in the thread - but it is really frustrating. It is impossible to have a conversation this way.

[neild]

A small counter-proposal:

A module may be deprecated by attaching a // Deprecated: comment to the module statement in go.mod. This follows the existing convention for deprecating packages and exported symbols via a doc comment.

// Deprecated: Use example.com/ancient/foo/v2. It's ever so much nicer.
module example.com/ancient/foo

The go tool will print the contents of deprecated comments at appropriate times.

$ go get -u
go: example.com/ancient/foo => v1.2.3
go: example.com/ancient/foo is deprecated:
go:   Use example.com/ancient/foo/v2. It's ever so much nicer.

This is more flexible than reporting on a new major version, since it permits also providing a notice suggesting an entirely different module path or none at all. ("Deprecated: This is full of security holes and will never be fixed.") For example, this would provide an simple way for golang.org/x/net/context to point users at context.

This easily permits the case of "proposal 2" here: Tag a version containing nothing but a go.mod with a // Deprecated: comment. Users fetching this version will be broken (as desired) and will receive an explanation as to why.

[elioengcomp]

@elioengcomp

There should be a way to get deprecated modules using regular Go tooling so Go Modules proxies can fetch and serve those versions as well.

I don't quite understand why. Module proxies may need to fetch deprecated/discontinued modules in order to learn about them, but they should never serve them, because a deprecated/discontinued module should fail to be selected by MVS, and fail to compile.

@peterbourgon

Module proxies do not know which version of Go is being used by clients. Enforcing these rules on the proxy side will provide a different behavior compared to what users get when they resolve from source and so can break the reproducibility of builds. The proxy should serve the content and let the client decide if it is good or not.

Users should be free to select which version of Go to use and when to upgrade so they should only be affected by the deprecation rules when they decide to move to a Go version that has it, regardless of where they are getting their modules from.

[peterbourgon]

@elioengcomp

Module proxies do not know which version of Go is being used by clients. Enforcing these rules on the proxy side will provide a different behavior compared to what users get when they resolve from source and so can break the reproducibility of builds. The proxy should serve the content and let the client decide if it is good or not.

I see, this makes sense 👍

[peterbourgon]

@neild

Tag a version containing nothing but a go.mod with a // Deprecated: comment. Users fetching this version will be broken (as desired) and will receive an explanation as to why.

Just so I understand fully: does that version break because the go tool detects the // Deprecated: tag, or because the module at that version doesn't contain buildable .go files?

[neild]

Just so I understand fully: does that version break because the go tool detects the // Deprecated: tag, or because the module at that version doesn't contain buildable .go files?

Because it contains no buildable .go files.

You can, of course, do this today. The only difference from "Proposal 2: A new deprecated directive" above is that the user won't be notified about why they're broken. Surfacing a deprecation notice from the go.mod would address that problem.

[peterbourgon]

@neild Got it. I need to think about it a bit more deeply, but on the surface, I like it.

[sylr]

@neild Might I suggest:

//go:deprecated Use example.com/ancient/foo/v2. It's ever so much nicer.
module example.com/ancient/foo

//go:... lines are known to trigger mechanisms. Simple // Deprecated: ... might bring confusion to people not aware that a simple comment can have build effects.

[Merovius]

@sylr The Deprecated comment has precedent. Whatever you think of it, I feel it's more confusing to have two different conventions.

[sylr]

@Merovius Even if "standardized", it remains a simple comment, it should not induce any building behaviour.

I think that if proposal #2 were to be implemented it should rely on either a directive or a pragma, not a comment.

[jimmyfrasche]

The nice thing about a directive with an optional import path in a sentinel release is that the go tool can follow the discontinuations/redirects until it finds the least version that's still maintained at the appropriate path.

If foo/v2 and foo/v3 are discontinued and v4 is at a new path entirely, bar/v4, the message for trying to upgrade v2 could see that its discontinued, look at foo/v3 and see that it's moved to bar/v4. If discontinuation is surfaced in the module proxy, this will be very fast.

If it doesn't follow this automatically, either

• the consumer needs to run go get successively until they find something that builds (or more likely just rummage around the docs a little)
• the producer needs to go back to add new discontinuations to all previously discontinued major versions every time a new major version is discontinued so they all point to the latest least supported major version

I don't really want to do either.

Having

// Deprecated: this is going away
module bye/bye

that's surfaced in some way by the go tool, like just printing it when upgrading to that version, would be a nice way to signal that discontinuation may be coming but even if it's not you should move on at some point.

[neild]

If foo/v2 and foo/v3 are discontinued and v4 is at a new path entirely, bar/v4, the message for trying to upgrade v2 could see that its discontinued, look at foo/v3 and see that it's moved to bar/v4. If discontinuation is surfaced in the module proxy, this will be very fast.

I would question whether this case is one worth optimizing for. How many modules have a lengthy sequence of major version increments, including jumps across entirely different module paths?

A deprecation notice is simple, and can be written in a forward-looking fashion that doesn't mention a specific updated version:

// Deprecated: Please find the most recent supported version of this module at http://example.org/"

I also wonder if the case of a module which goes through many incompatible API changes and offers no support for older versions would not be better addressed by remaining at v0.

[jimmyfrasche]

Probably not that many now but that number can only increase. Even if the rate of increase or at least relative proportion stays low, arguably it being uncommon would be a good argument for its UX being optimized: so that the uncommon situation is as pleasant and uniform as the common situation.

If github shuts down 10 years from now, I wouldn't want to have to read hundreds of bespoke deprecation notices to figure out where all my dependencies landed. But that could be a case against conflating deprecation, discontinuation, and redirection, because in that scenario the new import paths are likely compatible for all versions and simply written differently.

[bcmills]

As @neild notes, if you want to break go get -u for existing users (and go mod tidy for new users), you can do that today without adding any new syntax or semantics to the go.mod file. (However, I would second the observation from @kokes that bumping the PATCH part of the version when doing so is not at all semantically appropriate.)

Note that #24031 (already approved and under review, probably going to land in 1.16) would give you another such mechanism: you could publish a v1.7.3 that retracts everything in the range [v0.0.0-0, v1.7.3].

As far as I can tell, the non-redundant part of Proposal 2 (which really ought to have been filed as a separate proposal!) is the opportunity for the module author to suggest some specific replacement when they break the older major version. And I agree with @neild that that would be better-served — and more idiomatic — as a distinguished comment than as a new, redundant directive.

[bcmills]

So, let's turn to Proposal 1.

The specific text proposed is:

go: note: more recent major versions of github.com/peterbourgon/ff are available
go: note: to install the most recent one, run `go get github.com/peterbourgon/ff/v3`

Even ignoring the likelihood of incompatible API changes, the suggested command would not result in a working build: the user would still need to update their import paths to refer to the packages in the new module in order to use it, and if the packages include shared state and (as @peterbourgon suggests) the module author cannot be bothered to rewrite them in terms of the new API, then all other dependencies using that module may also need to be updated to use the new major version.

We have been trying to eliminate diagnostics from the go command that suggest commands that will not fix the user's problem. I certainly do not want to add more of them.

If there were some tool to automate the import-path rewriting (perhaps #32014 or #32816), then perhaps we could suggest that tool, but as it stands I think the suggested message needs to be rethought.