proposal: cmd/go: make major versions optional in import paths

[pkieltyka]

Semantic Import Versioning (SIV) is a novel idea for supporting multiple versions of a package within the same program. To my knowledge and experience, it's the first example of a strategy for supporting multiple versions of a project / dependency in the same application. More importantly though, its clever introductory design allowed it to offer multi-versioned packages in a Go program while maintaining Go's compatibility guarantee.

Multi-versioned packages in a single program can be quite powerful -- for instance, imagine a Web service where you'd like to maintain backwards compatible support for your consumers, you can simply use an older import path from the same origin repository and versioning control, and quite elegantly continue to support those older API versions.

Although SIV may be an elegant solution in the scenario described above, it also adds unnecessary complexity, cost, code noise, discoverability and ergonomics for the majority of packages (publicly and privately) which may not ever have a mutli-version requirement (I'd argue most packages, and simply we can look to other ecosystems to see this is true). I am sure the Go team has heard a lot of feedback on the friction of SIV. https://twitter.com/peterbourgon/status/1236657048714182657?s=21 and https://peter.bourgon.org/blog/2020/09/14/siv-is-unsound.html offers some excellent points as well.

Clearly there is a case for SIV as an elegant solution for supporting multiple versions of a package in a single application, and there is also a strong case to make SIV optional.

It's clear to me there is a design trade-off at hand, and there is no single correct answer. As I consider the 80/20 rule in making an architectural decision between two trade-offs of capability and usability, I prefer to go with the 80% case so long as it doesn't forego the 20% ability. Which design is the best to optimize for if we can still support both? In the case with Go today, its not possible to opt-out of SIV, or opt-into SIV -- I believe both approaches can yield a happy solution. If we were starting from the beginning, I'd suggest to have SIV be opt-in, but maybe at this point its better for it to be an opt-out design to maintain backwards compatibility with history.

---

I'd like to propose a path to make SIV opt-out at the level of an application developer consuming a package, while being backwards compatible with current packages and tools.

I'd like to use https://github.com/go-chi/chi as an example for this proposal which adopted semver ahead of Go modules and SIV, and is built for developer simplicity and ergonomics intended for pro Go developers, but also making it familiar and accessible for developers who are new to Go -- these are my design goals for chi as an author and maintainer as started back in 2017. My present goal is to release Chi v5 without a SIV requirement and the only way I can do so is with the proposal below:

---

Proposal, by example:

github.com/go-chi/chi/go.mod:

module github.com/go-chi/chi/v5

go 1.16

then, git tag chi as v5.0.0 to make the release.

Application developers may consume the package via go get github.com/go-chi/chi@latest or with @v5 or @v5.0.0 and the expected import path will be "github.com/go-chi/chi", however "github.com/go-chi/chi/v5" import path would also be valid and usable.

In the above case, we're specifying the go.mod as expected with current behaviour with SIV from a library perspective. However, from the application perspective when fetching or consuming the library, I may opt-out of the "/v5" suffix in the import path and only adopt it in the scenario when I'd like to support "/v5" and "/v4" (or some other prior version), where I require the handling of multiple versions simultaneously in my program.

I believe the implementation of the above to be backwards compatible as developers would continue to use "github.com/go-chi/chi/v5" with older version of Go as SIV is implied, but optionally developers could make their choice of multiple-version support for the package by handling the import paths themselves and import "github.com/go-chi/chi" to utilize v5.x.x as specified by go.mod.

I believe changes to the Go toolchain for such support would be minimal and would be isolated to the components which build module lists.

Thank you for reading my proposal, and its consideration.

[peterbourgon]

Application developers may consume the package via go get github.com/go-chi/chi@latest

I believe this would represent a breaking change, as it is currently (?) parsed as the latest minor.patch version of major version v0 or v1. That's an artifact of what is I think the major problem with any form of optional SIV: the unversioned module path is, unfortunately, interpreted not as "no version specified" but instead as major version 0 or 1.

[theckman]

@peterbourgon I'm not sure the Go toolchain has ever been covered under the compatibility guarantee, based on other "breaking" changes that have been made. The guarantee explicitly says

Compatibility is at the source level.

[theckman]

I'm currently a maintainer of the https://github.com/PagerDuty/go-pagerduty package, which was also incepted before Modules and never had 0.x releases. This decision for not using 0.x was made because they wanted to commit to not breaking the current version of the API, because PagerDuty is a critical service, while relying on the ease of major version bumps to have consumers pick up breaking changes as they get made within the library. This was the way they were going to iterate going forward, so that folks could lock into "stable" versions because they didn't want pulling in a 0.x release subtly breaking a consumer.

As of now I'm planning a v1.5.0 breaking release to work around SIV not being optional. There is a bug that makes the current API not work reliably (two conflicting fields trying to be unmarshaled into), and removing the erroneous field may result in some people needing to update their code to use the .ID field instead of .Id.

I cannot in good faith justify forcing people to update all of their files that use our Go Module with the SIV for a one character change that they may use in one file. I'm sorta justifying the breaking change to myself because the API has never been stable due to the conflicting field names.

It would be ideal if the SIV component of the import was optional, if only one major version of a dependency is present in the go.mod file. That way folks could update their dependency, and only update the actual files that need to be changed. Likewise, we are still able to support more complex projects who need to roman-ride the major versions while doing the transition.

I genuinely believe it would be a great idea for us to really consider this proposal, and that it would be a nice improvement to the user experience of the Go Toolchain and Modules. I am confident that the need to roman-ride between two major versions of a package will only be needed by a small subset of the larger Go Ecosystem, and so I wonder if it makes sense to make it a requirement for all. I believe accepting this proposal would decrease the cognitive hurdles that may need to be overcome when trying maintain a Module, while also giving us the ability to be flexible for those who really need it.

Edit: Instead of reacting with :-1: this comment, could you comment below quoting what you disagree with and why? I think that would result in a much more constructive interaction.

[theckman]

@nbys I see you 👎 on my comment. I'm sorta gonna put you on the spot and ask if you can explain what part of that you disagree with and why you disagree with it?

[peterbourgon]

I am confident that the need to roman-ride between two major versions of a package will only be needed by a small subset of the larger Go ecosystem . . .

And even for that tiny fraction of the ecosystem, the requirement typically exists only transiently, during a dependency upgrade process that requires multiple steps. And this might be worth stating explicitly, because I'm not sure it's understood by all the stakeholders: codebases which need to allow multiple major versions of a dependency in a single compilation unit in order to make a complicated upgrade tractable are pathological, not normal. Getting into that state isn't an inevitable outcome of writing software, it's a product of a specific set of conditions representing a super-minority of projects in the overall ecosystem.

[thrawn01]

I'm the maintainer of https://github.com/mailgun/mailgun-go and I do understand this pain. The process of updating the mailgun-go library to v4 involved not only updating all the import paths but also every single golang code snippet in the mailgun documentation. This was a huge under taking that we don't wish to do again.

Our hope is that https://github.com/golang/go/issues/32014 will get approved and standard tooling will exist to make upgrading import paths simple for both library devs and users. While this doesn't solve the need to update all our example snippets it should help ease the some of the pain that library maintainers have in this manner.

[pkieltyka]

Respectfully, https://github.com/golang/go/issues/32014 is not a real solution to solving the challenges with SIV. As you said yourself, all of your documentation had to change as well, and that is just the beginning of the permanent tax on developer experience that SIV imposes for a niche use-case of multi-version support in a single program.

[bcmills]

@theckman, I commented on your specific example in https://github.com/PagerDuty/go-pagerduty/issues/218#issuecomment-784486255.

I'm somewhat skeptical that a breaking change is really necessary there, but even if it is, it seems like the sort of fix that would be allowed within a major version under a Go-1-style compatibility policy. So I personally don't find that example particularly compelling.

[theckman]

@bcmills The latter part of that post was showing a case where we ran into that problem that would only be a one-character change in consumer projects, but the SIV would make it so much larger. There is also this, which is a similar class of issue https://github.com/PagerDuty/go-pagerduty/pull/251 which I don't believe can be easily resolved without a BC. Let's focus on this overall need / desire instead:

This decision for not using 0.x was made because they wanted to commit to not breaking the current version of the API, because PagerDuty is a critical service, while relying on the ease of major version bumps to have consumers pick up breaking changes as they get made within the library. This was the way they were going to iterate going forward, so that folks could lock into "stable" versions because they didn't want pulling in a 0.x release subtly breaking a consumer.

How would you address this need in Modules?

[ulikunitz]

Making it costly to break compatibility is a feature not a bug.

@rsc has explained it here: https://research.swtch.com/vgo-import. I don't agree with Russ on everything (#38776) but I agree with him here.

Breaking changes are bad. Who is using Perl 6? Python 3 was a pain. Successful software usually takes compatibility seriously. Windows fakes internal structures to allow major applications still to run.

Linus Torvalds wrote this:

Breaking user programs simply isn't acceptable. (…) We know that people use old binaries for years and years, and that making a new release doesn't mean that you can just throw that out. You can trust us.

I may also may remind on the Go 1 compatibility guideline was a huge factor for its success. You might want to read it again: https://golang.org/doc/go1compat

So if compatibility is such a huge factor for software success, why want to make it easy to make incompatible changes?

[nbys]

@nbys I see you 👎 on my comment. I'm sorta gonna put you on the spot and ask if you can explain what part of that you disagree with and why you disagree with it?

I am sorry if my downvote in some way offended you.

I cannot in good faith justify forcing people to update all of their files that use our Go Module with the SIV for a one character change that they may use in one file. I'm sorta justifying the breaking change to myself because the API has never been stable due to the conflicting field names.

I could add my cents only from a library-user perspective. I do not think of the necessity to change imports for a new major version as something bad. SIV enforces us to consider an update to the major version as a serious change. In my opinion, it is worth to grep the project, change the import paths and actually revisit library usage. The proposal won't change anything for me. But I believe @latest will be then a common thing in a lot of libraries. And only because it is just easier.

P.S. sorry for my English

[theckman]

@nbys absolutely no offense, a :-1: just doesn't help me challenge the ideas / opinions I have. Thank you so much for taking the time to reply so that I can understand where you're coming from. 👍

P.S. Your English was great!

[peterbourgon]

Making it costly to break compatibility is a feature not a bug.

The cost of breaking compatibility isn't constant, it's a function of many variables that can be different from project to project.

Breaking changes are bad.

Not universally. Breaking changes are sometimes necessary, or even good.

[deltamualpha]

I have a question about a specific part of the proposal from an application developer POV:

Application developers may consume the package via go get github.com/go-chi/chi@latest or with @v5 or @v5.0.0 and the expected import path will be "github.com/go-chi/chi", however "github.com/go-chi/chi/v5" import path would also be valid and usable.

I import and use "github.com/go-chi/chi" and get version 5. (Assume, for the sake of argument, that no other libraries I use affect which version of chi MVS resolves to.) The chi developers tag and release v6.0.0. I install a new library, and run go mod tidy. Do I now find myself depending upon chi v6?

[pkieltyka]

@deltamualpha no, upgrading to major versions should not be implicit via go mod tidy. Once a module in go.mod is set with a specific major version (similar to how every other package manager works in other ecosystems), a developer must explicitly instruct the module system to upgrade to a new major version.

@latest would imply the current major version which would be recorded in go.mod as the latest major at the time. In order to upgrade to v6.0.0 one would do go get -u github.com/go-chi/chi@v6.0.0 or go get -u github.com/go-chi/chi/v6. However, I would argue that if one calls go get -u github.com/go-chi/chi@latest after a go.mod is set, then in this case it would also upgrade to the latest major release + version, in your example as v6.0.0.

However, the above notes are easily debatable and I don't hold any strong opinions on version management (other then of course go mod tidy should certainly not implicitly upgrade to a major version if one is already set in a go.mod).

[bcmills]

@theckman

This decision for not using 0.x was made because they wanted to commit to not breaking the current version of the API, because PagerDuty is a critical service, while relying on the ease of major version bumps to have consumers pick up breaking changes as they get made within the library. This was the way they were going to iterate going forward, so that folks could lock into "stable" versions because they didn't want pulling in a 0.x release subtly breaking a consumer.

How would you address this need in Modules?

Honestly, I would not “[rely] on the ease of major version bumps” at all. If a downstream user needs a fix for bug, and that fix is after a breaking change, then their only options are to either backport the fix to some fork of the package (possibly a release branch), or to stop and upgrade their code before they can fix it.

Instead, for the few cases where existing, previously-supported API is unsalvageable (such as the erroneous Id field, and perhaps the erroneous Targets fields in https://github.com/PagerDuty/go-pagerduty/pull/251), I would rely more heavily on deprecation and/or compatibility shims (such as {Marshal,Unmarshal}JSON methods, for those examples).

[bcmills]

(On a bit of a side-note, someone in the community recently showed me Rich Hickey's excellent Spec-ulation talk, which goes into great detail about the various dimensions of compatibility, versioning, and namespaces. It's worth a watch!)

[D1CED]

Breaking changes are bad.

Not universally. Breaking changes are sometimes necessary, or even good.

This debate is not about ever making a breaking change but frequency and scope of them.

One camp is in favor of low frequency large breaking changes and the other prefers more frequent and small breaking changes.

This proposal is a complaint of the second camp that they have to edit all imports on a release of a new major version to upgrade.

Either approach has its advantages and disadvantages but the community has to decide what to encourage/discourage.

[peterbourgon]

This debate is not about ever making a breaking change but frequency and scope of them.

Yes.

One camp is in favor of low frequency large breaking changes and the other prefers more frequent and small breaking changes.

No. One "camp" (i.e. Go modules) asserts that breaking changes are always very costly and should always be avoided. The other "camp" (i.e. myself and others) observes that breaking changes are not always very costly and do not necessarily need to be avoided.

Either approach has its advantages and disadvantages but the community has to decide what to encourage/discourage.

Go the language can take very opinionated stances on a wide variety of topics, because it's just one programming language among many. If any of its opinions are a problem for a potential user, that user can simply opt out and not become a Gopher. But Go modules doesn't have the same license to assert what is and isn't allowed, because it has a monopoly on package management in the Go ecosystem. If any of its opinions are a problem for a potential user, that user cannot reasonably opt out and use another tool. By and large, modules, and other mandatory tooling, has to meet users where they are, not direct them to where the authors feel they ought to be.

[ulikunitz]

Thinking further about it: Go's target was it to make software work at scale. And at scale every breaking change creates huge costs due to the number of imports of a module. Semantic import versioning actually reduces the cost of a breaking change at scale by giving it another name. And if you publish your code on the Internet you are working at scale, if you want it or not.

@bcmills Thank you for providing the link to Rich Hickey's talk. I can add him now to the people that think compatibility is a factor for successful software.

[theckman]

And at scale every breaking change creates huge costs due to the number of imports of a module.

@ulikunitz what if 99.99% of those who import you don't use the functionality you're breaking in the major version change, and as such it's a no-op to upgrade to your new major version. Wouldn't it be a much larger cost to force everyone to update their imports?

[peterbourgon]

Thinking further about it: Go's target was it to make software work at scale. And at scale every breaking change creates huge costs due to the number of imports of a module . . . And if you publish your code on the Internet you are working at scale, if you want it or not.

Not all Go modules get published to the public internet. Many are kept private, in corporate or other organizations.

Not all modules are widely imported. Many are used by just one, or just two, consumers — especially private modules.

Not all modules, even widely imported modules, create huge costs when they make a breaking change. Consumers pin to a version and use it without interruption, regardless of the rate of iteration on the major (or minor, or patch) version numbers.

And not all modules contain code that should work at scale, in the sense that you mean. Go is a general-purpose programming language, suitable for more than one type of user.

These are all normative claims about how the Go ecosystem should be, according to some specific (i.e. non-universal) set of assumptions. They don't describe the Go ecosystem as it actually exists.

[peterbourgon]

@ulikunitz what if 99.99% of those who import you don't use the functionality you're breaking in the major version change, and as such it's a no-op to upgrade to your new major version. Wouldn't it be a much larger cost to force everyone to update their imports?

Well, hopefully the goal here is to make it easier to consume major version updates, not to make it easier to violate semver 😉

[theckman]

Well, hopefully the goal here is to make it easier to consume major version updates, not to make it easier to violate semver 😉

I thought that's what I was communicating, making the cost to upgrade proportional to the change needed. 🤔

[bytheway]

I'm just a go user, but I really liked the idea of SIV. I feel like I understood the benefits, even if there was some pain.

Now that it is mandatory, however, I've watched public, useful libraries actively avoiding incrementing the major version when it would be the natural/semver thing to do, instead opting to rev a minor version number and "break" with semver guidelines.

We've faced similar challenges and decisions with internal libraries at my company. What could be a quick communication about an incompatibility and a one-line update to the go.mod to get the latest package can quickly turn into a large change set, touching many, many files.

I would love an option to opt-in to SIV when appropriate, with a less invasive default which doesn't require large changes to codebases, documentation, etc. in both the the package and importers of the package when a major version is bumped.

Whatever the benefits of SIV are in theory, they are being rendered moot by the practice of library authors. I would love to see mandatory SIV revisited.

[veqryn]

I actually like SIV. I just wish it wasn't so hard to use SIV and so costly for small projects. If I could go back in time, I would have forced all v0 and v1 modules to have that in their module path. Such as:

module github.com/veqryn/someproject/v0

go 1.16

and

module github.com/veqryn/someotherproject/v1

go 1.16

and

module github.com/veqryn/somematureproject/v2

go 1.16

Then, a small project might require these repos:

module github.com/veqryn/smallproject

go 1.16

require (
    github.com/veqryn/someproject/v0 v0.1.5
    github.com/veqryn/someotherproject/v1 v1.80.234
    github.com/veqryn/somematureproject/v2 v2.0.0
)

And then in the small project's code, they could use imports without the "vX" in them, so long as their go.mod never had multiple versions of that repo:

package smallproject

import (
    "github.com/veqryn/someproject"
    "github.com/veqryn/someotherproject"
    "github.com/veqryn/somematureproject"
)

However, as soon as a project (or its dependencies) use more than one version, it would be suddenly required to add the "vX" to its imports:

module github.com/veqryn/bigproject

go 1.16

require (
    github.com/veqryn/somematureproject/v1 v1.9.1
    github.com/veqryn/somematureproject/v2 v2.0.1
)

Required to add the vX to the path, or else compiler error:

package bigprojectold

import "github.com/veqryn/somematureproject/v1"

and

package bigprojectnew

import "github.com/veqryn/somematureproject/v2"

If a dependency uses (requires) github.com/veqryn/somematureproject/v2, and your project uses (requires) github.com/veqryn/somematureproject/v1, then they both end up in your project's mod file, and you would be forced in your project to specify the "v1" path on all imports in your project. The go compiler would be smart enough when compiling your dependencies to know that if the dependency only requires one version in its own mod file, then any imports in that dependency would use that version in the dependency's mod file, even if they don't have the "vX" in them.

I feel like this would be similar to how "embedding" structs in Golang lets you call member's function, until you have a conflict between multiple members with the same function name, and then Golang forces you to declare the function in the parent to resolve the conflict.

I think this would have solved a lot of problems, but I also don't see how to get there in an easily backwards compatible way. Go mod would have to declare all projects missing v0 and v1 in their module name as "incompatible", and have special logic for dealing with them.

[beoran]

How about, in stead of this proposal, adding a go mod upgrade or such which installs the new major version of the dependency but also automatically updates all related imports in all go files in the module? Like that some of the pain of upgrading can be lessened already, in cases where the API didn't change all that much.

[Merovius]

I am confident that the need to roman-ride between two major versions of a package will only be needed by a small subset of the larger Go ecosystem . . .

I do not understand this confidence. As far as I can tell, the need arises for any module a) that has a diamond in its dependency graph and b) where the bottom of that diamond bumps their major version at some point. I do not understand why you think that would be a small subset of the Go ecosystem. Alternatively, you don't think those criteria are sufficient to imply the need, which seems entirely non-obvious to me as well - personally, I consider the argument Russ makes to be very convincing.

[Merovius]

@pkieltyka In the proposal text, you distinguish between "the application perspective" and "the library perspective". As far as I'm aware, the only meaning of this in the Go ecosystem is "a package main" and "any other package" - but ISTM that's unlikely to be the meaning you intend (for one, most Go modules containing a package main also contain other packages. Also I can't imagine that your proposal really only targets imports written in a package main). I think it would be helpful if you could explain what you mean by that.

Note, in particular, that the author of the main module often has little control over whether or not they need multiple major versions in their package - if any part of their dependency graph requires it, they do too. So a priori, I think, there are more than two perspectives - you have leaf modules, intermediate modules (that both have dependencies and are dependencies) and main modules and each of those might or might not specify major versions in their import paths. And we need to consider how a major version bump in any of those cases affects the others. That matrix can probably be reasonably simplified down (for example, we probably only need to worry about major version bumps in leaves), but all these cases need to be covered somehow.

For example, say we have a leaf module A which module B and C depend on. Say B and C have opted out of SIV and want to use v2 and v3 of respectively. Now a fourth module D wants to import both B and C. Is that simply impossible, until they convince the authors of B and C to switch to SIV (or convinceB to move to A@v3)? What major version would D use? Assume A declares a type T and C type-assert an argument to a function on A.T - if D passes an A.D from B to C, does that type-assertion fail?

Note that even if we accept "it only affects a minority of Go programs", that still doesn't absolve us from specifying the semantics when it does. I think the proposal needs a bit of elaboration on that front. Currently, an import path uniquely identifies a package, in a single build. Under your proposal, we both have a) different import paths referring to the same package and b) the same import path referring to different packages. It's not super clear to me how that would work, in respect to changes to the compiler and linker and reflect, for example. One appeal of SIV as a solution is that it's transparent to the language - the compiler doesn't have to know about modules and major versions, it just sees packages in opaque import paths.

[peterbourgon]

I do not understand why you think that would be a small subset of the Go ecosystem.

I have personally had what I consider to be substantial exposure to an enormous amount of Go code, due to my position in the OSS ecosystem, as well as my consulting work. That exposure includes, importantly, a huge amount of code maintained in private repositories. I have no way of knowing, but I suspect the only person in the Go community who may have seen more Go code than I have is Bill Kennedy. With that context, I can state without hesitation that the need to include two major versions of the same dependency in one compilation unit is extraordinarily rare, and in those rare circumstances that it does arise, it is almost always due to pathological conditions in the dependency graph.

[Merovius]

@peterbourgon I'm not saying what you say about your expertise is false, but it's still an appeal to authority and not an actual argument. Instead of responding to the arguments as to why the need exists or pointing out the wrong assumptions made, you are simply asserting that the need doesn't exist. I feel I was very specific. As far as I can tell,

1. The need arises in any project that has a diamond in its dependency graph, where the leaf bumps a major version and
2. This seems very normal and common

The only way I can interpret your comment as actually addressing the arguments made in favor of SIV is that you consider diamonds to be "pathological conditions in the dependency graph", which I find extremely surprising and worthy of at least some data.

[peterbourgon]

Yes, it is an appeal to authority, unfortunately, because I'm not aware of how better to make the point. It's not possible to get representative data on how Go exists in the wild, because the vast majority of it — ca. 80%, if common estimates are to be believed — is private.

[Merovius]

@peterbourgon You could make a start by saying if you think 1., or 2., or both is wrong - even if you can't provide statistical data to prove that.

FTR, you probably have seen more Go code in the wild than me. If you say "diamonds in dependency graphs are rare", I'd be willing to believe you, even if I find that implausible. Same with "diamonds are common, but leafs rarely bump their major versions". If, based on your expertise, you could make an argument why even in the presence of diamonds with leafs with multiple major versions, there still is no need to support having both in one program (for example, if you could provide a solution for the author of D in the scenario I sketched above), I might be convinced that there really is no need (or try to convince you that there is).

Your assertion "there is no need" doesn't let me distinguish between a) my assumptions about Go code in the wild are wrong (as I haven't seen enough of it) or b) my deduction about when SIV is needed are wrong (as I've overlooked an alternative solution) or c) your assertions about Go code in the wild are wrong or d) your deductions about what set of go code needs SIV is wrong.

As long as you simply assert that in practice, supporting multiple versions isn't needed, we are at an impasse. You won't be able to convince anyone that SIV is broken and no one could ever convince you that it isn't. And we are doomed to forever have the same conversation.

So again: Is it 1. the conclusion that in that situation, you need multiple major versions in one program, or 2. the assumption that diamonds with major version bumps in the leaf are common, which is wrong? If 1, what is wrong about it?

[edit: Made the numbering consistent, to avoid confusion in a potential answer]

[DmitriyMV]

It was already said, that if you don't want to have backward compatibility, or support legacy code, you just stay in v0.x.y.

[myitcv]

It's not possible to get representative data on how Go exists in the wild, because the vast majority of it — ca. 80%, if common estimates are to be believed — is private.

In my experience of working with private code bases and clients, I disagree - it is possible to get representative data. Private examples can, and should, be distilled into anonymised examples. This can either be done by humans, or better by tools.

[pkieltyka]

@Merovius

In the proposal text, you distinguish between "the application perspective" and "the library perspective". As far as I'm aware, the only meaning of this in the Go ecosystem is "a package main" and "any other package"

The two high-level perspectives I'm referring to for a module: a module's go.mod file as its module definition, and from an application which is consuming that module.

Note, in particular, that the author of the main module often has little control over whether or not they need multiple major versions in their package

In situations where this is true -- as my proposal above outlines -- SIV is still valid and compatible, and library authors may continue to leverage the SIV feature for an upgrade path where that is appropriate. As you mentioned further, "That matrix can probably be reasonably simplified down (for example, we probably only need to worry about major version bumps in leaves), but all these cases need to be covered somehow." -- I agree, and this is the way.

---

I do think however, as @peterbourgon has explained, and I will similarly echo, in practice this (multi-version deps in the same program) rarely happens, and it also never happens in other ecosystems. Go makes it possible for the niche scenarios which is wonderful and innovative, but for anyone to suggest it should be mandatory (as effectively it is now), you must take a step back and consider the developer experience from all perspectives. Clearly @peterbourgon and I care a lot about Go, its community and module system -- otherwise we'd just accept the status quo and not care to do better. We certainly can do better as a community and IMHO, the priority and focal point should be with the developer experience. Please lets remember the roots of the Go language as Rob Pike had intended, and what makes the Go language so wonderful, and that is -- it's simplicity. Simplicity is Complicated -- yes, but its worth it.

Please consider how other developer ecosystems have been working for decades and continue to work into the future. Please consider the growth of Go, its trajectory of adoption and how new developers are trained or gravitate to Go, and how our module system will be familiar, appreciated or confusing to them. I've personally trained dozens of developers in Go on my teams, and I can let you know, Go modules continue to be the primary source of pain. We're discussing a niche feature of Go (SIV) that enables multi-versions of a dependency in a single program as a panacea to dependency resolution, when others who maintain large OSS projects in the community have experienced it as a source of friction, regression and fracture to the simplicity of Go's developer experience -- this should be taken very seriously.

Finally, as I mentioned in my original proposal text, we must look at this design trade-off from an 80/20 rule perspective and consider what is the 80% use case of modules / dependencies? Is it for a single version of a dependency in a program, or is it multi-versions of a dependency in a program? (The answer is clearly "single version"). We must always design our systems in favor of the 80% rule, and most especially when it relates to the simplicity of the developer experience. My proposal is intended to get us on the right track while still maintaining backwards-compatible support of SIV, while also simplifying the developer experience.

[Merovius]

@pkieltyka

The two high-level perspectives I'm referring to for a module: a module's go.mod file as its module definition, and from an application which is consuming that module.

What about non-applications consuming that module? Obviously, if you only consider dependency graphs that have a main module and one layer of dependencies, you won't run into a lot of version-management problem. But that's orders of magnitude rarer than even dependency graphs without diamonds.

I agree, and this is the way.

What is the way? Naive combinatorics already give 18 different combinations that I mention and you specifically only talk about two of them, without explaining why that would cover more.

Finally, as I mentioned in my original proposal text, we must look at this design trade-off from an 80/20 rule perspective and consider what is the 80% use case of modules / dependencies?

But we are currently disagreeing about which one is the 80% case.

I am trying to make a case as to why I think the overwhelming majority of Go programs do need the capability of having multiple versions of the same module. And in the interest of reconciling this difference of opinion, I'm trying to make it as simple as possible to understand and as easy as possible to refute. Just tell me if you think 1. you can solve the situation where the leaf of a diamond bumps a major version, without requiring to compile in multiple versions (and if so, how that solution looks) or 2. you think that diamonds in dependency graphs almost never exist in practice or the leaf almost never bumps their major version.

It's a simple choice - 1, 2, both, or neither. It does not help to repeat "in practice, most Go programs don't need multiple versions", because that doesn't tell me if we are disagreeing what it means to need them, or if we are disagreeing about how common that situation is.

Moreover, even if we put this particular disagreement aside for now and just accept the claim that multiple major versions are rarely needed, there are still open questions that need answering to understand what you are proposing. These questions are very concrete and the answer doesn't have to consist of more than a "yes" or a "no" for many of them. But I, at least, need an answer, if I want to understand what you are proposing exactly and thus, if it's a good idea. And I'm sorry, but I specifically said that just saying "it's rarely needed in practice" doesn't help, because we still need to actually decide what happens, if we want to implement it.

Just to illustrate, one of the very specific, yes or no questions I asked is

Is that simply impossible, until they convince the authors of B and C to switch to SIV (or convince B to move to A@v3)?

You might very well say "yes, that is impossible". You might also say "no, it would still be possible, they could do X". It's a yes-or-no question and either answer would be helpful, just to understand what the actual implications of your proposal are. We first need to know what the costs and the benefits of your proposal are, before we can form an opinion about if the cost is low enough or paid infrequent enough, to not outweigh the benefits.

Of course, you don't have to answer to me. I'm only speaking for myself. But I assure you that I'm really just trying to be open to your ideas here and I'm trying to clarify what, in my opinion, is needed to realize them.

[rogpeppe]

I am trying to make a case as to why I think the overwhelming majority of Go programs do need the capability of having multiple versions of the same module.

FWIW I don't think this issue has to be about the capability of having multiple major versions of same module. I think that's a done deal at this point.

I do think however that it's possible to consider the possibility that the major version of a module might not need to be specified in every import path. When a module is only directly dependent on a single major version (I think no-one is disputing that this is the most common case), the major version could be implied from the major version that's specified in the go.mod file.

The original argument against this when I first brought this up was that an important property of Go is that all identical import paths within a build refer to the same package. I'm not entirely convinced that this property is that important, and the pain of updating import paths is very real, so perhaps we could reconsider that decision. The other argument was that it was actually a good thing to make it hard to update major versions, but I'm not convinced about that either.

Because it's relevant, I'll repeat my original suggestion here:

When an import path within a repo does not contain an explicit major version, that version is implied by looking at the go.mod file for that repo?

Some possible rules:

• select all modules from go.mod that match the repo name.
• if there's more than one, error with "ambiguous import path"
• otherwise select that major version.

[pkieltyka]

FWIW I don't think this issue has to be about the capability of having multiple major versions of same module. I think that's a done deal at this point.

I do think however that it's possible to consider the possibility that the major version of a module might not need to be specified in every import path. When a module is only directly dependent on a single major version (I think no-one is disputing that this is the most common case), the major version could be implied from the major version that's specified in the go.mod file.

yeay -- 100% exactly. I hope my proposal conveyed that, but thank you for re-iterating it, and great to see your ideas dating back to 2018 on the same topic.

The original argument against this when I first brought this up was that an important property of Go is that all identical import paths within a build refer to the same package. I'm not entirely convinced that this property is that important, and the pain of updating import paths is very real, so perhaps we could reconsider that decision.

100% agree, and precisely why I shine light on 80/20 rule when making design decisions, favoring the developer experience while still achieving compatibility with what stands in SIV and module resolution. The pain is not worth requiring all identical import paths when the module resolution system can easily infer it when there is just a single major version of that dependency (which is the majority of the time).

[peterbourgon]

@myitcv

In my experience of working with private code bases and clients, I disagree - it is possible to get representative data.

I unfortunately disagree. Organizations willing to offer such data, even anonymized, are a nonrepresentative minority of the overall group.

[Merovius]

The original argument against this when I first brought this up was that an important property of Go is that all identical import paths within a build refer to the same package. I'm not entirely convinced that this property is that important […]

From a pure language-user perspective, I do like this property and I think losing it might make debugging harder in certain cases. I also think the required extra step of having to look up the major version used in the go.mod file (e.g. when trying to look at the documentation of a used function you don't know) might sometimes be confusing and/or frustrating. All of this might be worth it, though. Just putting the downsides on the table.

I think there are some implementation-specifics to be ironed out. I think it might be possible/sensible to canonicalize import paths - that is, insert a version specifier, even if the import doesn't include it. I don't know if that has knock-on effects, though.

But, yes, I don't have very strong objections to doing this, as long as it can be made to work. (Not to be petulant, but I did bring up this property above specifically to learn what @pkieltyka intends to do about it. "Drop it, we can make it work by […]" would've been a satisfying answer).

---

[edit] One quick addendum: "make it work", for me, also implies "does not affect any other module". If a module opting out of SIV means that any module requiring it has to deal with the consequences, I would strongly object. Again, that's why I asked for specifics of how any of this is done, it doesn't seem obvious to me, that it can't have knock-on effects.

[PaluMacil]

@theckman @pkieltyka I personally don't care deeply that SIV allows use of multiple versions at once. To me, the feature is the "code noise". It took a while for me to understand it this way, but I got to like it when I did my first major version upgrade after SIV. I upgraded badgerdb, and I was thinking I'd have a lot of paths to replace. However, I noticed immediately that it was only two files that changed because my repository logic called on my driver through an interface. Looking at places I use Gorilla Mux, I don't abstract over it with an interface, but I set all my routes in one files, so it would again be only in a couple files if I upgraded. If I was using a logger or database package all over and I ran into this issue, it might be a good place to abstract over this so that my tests can catch breaks and I can make fixes in one small place.

All that said, the frustration and confusion in the community would have led me to lean against SIV had it not already happened. Since it's in the past, I consider changing course equally bad to having some of the current confusion and frustration. Therefore, I would prefer not to see a change. I'd be fine with a go get flag people can use to update their imports upon upgrade, but I think enough tools (sed, IDEs, find and replace) do that now that it isn't the big issue for people compared to code noise.

[pkieltyka]

@PaluMacil thanks, and that opinion is well understood. I'm not proposing a breaking change, by my proposal, you may continue to use all the SIV you want and all the tools just as you are :)

[pkieltyka]

@Merovius I appreciate your points and you digging into the technicals of the proposal. I think many permutations and answers could be acceptable that satisfy the high-level direction I've originally proposed. I'd be curious to first hear from the Go team on alignment of the high level goals, and then take their direction on the best implementation forward. As @rogpeppe had mentioned, there were early decisions made during an early stage of Go's module existence / use / adoption, and today in 2021 we've learnt a lot since then and we should reconsider some of the early decisions, especially as there are approaches as Roger has proposed in the past, and I'm proposing now, to remove some of the pains and keep bw-compat with the module system.

The diamond problem is a classic one, but I still haven't determined to what extent (if any) my proposal will impact Go's module system in relation to the diamond problem. I still need to think more about it as I'm not precisely clear on certain edges of the module linking system, which is where I defer to other experts and the module system authors.

Here are two starting diagrams that hopefully clarify how I'm picturing the module definitions to look:

[PaluMacil]

@PaluMacil thanks, and that opinion is well understood. I'm not proposing a breaking change, by my proposal, you may continue to use all the SIV you want and all the tools just as you are :)

@pkieltyka I see better now. I had been mistaken in my assessment of your approach, but with emphasizing the consumer being unaffected, that seems like it would quell a lot of concern in the community. Appealing to a wide audience only need be balanced against the concept that more settings means more complexity for a newcomer to Go tooling. This changes the conversation to where I am probably happy either way.

[D1CED]

For those who take issue with having to increment the import version when upgrading a dependency I can offer a partial solution (aka hack):

You can use a replace directive to create a single name to import a module. Upon upgrade you would just need to update the replace directive.

Example:

replace echo => github.com/labstack/echo/v4 v4.2.0

Now you can import github.com/labstack/echo/v4 as echo.

The short coming is that this breaks for consumers of your module (so if it's a library) . They would need the same replace directive as in your module file. This has also the issue of name clashes and hence dependency confusion. So make sure your replacement does not look like an URL or is one that you control.

Quick proposal draft: Add a new alias module directive that allows one to use the alias in place of the actual module name. This alias would be free of SIV semantics and module local so that two aliases of the same name in different module would not conflict.

There's no problem in Computer Science that can't be solved by adding another layer of abstraction to it

[bcmills]

@beoran

How about, in stead of this proposal, adding a go mod upgrade or such which installs the new major version of the dependency but also automatically updates all related imports in all go files in the module?

That is more-or-less exactly #32014, and closely related to #32816,

[ianlancetaylor]

It is essential that Go executables use distinct import paths for distinct packages. That is not optional.

But it is not essential that the import path written in a Go file be the actual import path. It would be possible for cmd/go to decide that, within a given package/module, all imports of "frobozz.gopher/leftpad" are actually imports of "frobozz.gopher/leftpad/v73".

This would clearly raise the possibility of massive confusion when looking at source code. And I don't understand what is supposed to happen if the module is a library imported by a main package that also imports other library packages that themselves import "frobozz.gopher/leftpad".

[pkieltyka]

@ianlancetaylor @bcmills thank you both for taking the time to lend your expertise.

It is essential that Go executables use distinct import paths for distinct packages. That is not optional.

Makes sense. And IMHO, it checks the boxes for me on DX (developer experience) at this level, and is preferred. It would be nice however if go get github.com/pkg/app@latest could resolve to the latest major version when installing a program, but I don't hold any strong opinions here and just mentioning as food for thought.

But it is not essential that the import path written in a Go file be the actual import path. It would be possible for cmd/go to decide that, within a given package/module, all imports of "frobozz.gopher/leftpad" are actually imports of "frobozz.gopher/leftpad/v73".

Thank you for acknowledging this.

This would clearly raise the possibility of massive confusion when looking at source code.

I believe in the majority of cases from a developer perspective (in the context of their application), is that they're dealing with single major versions of a dependency. The app's dep graph likely contains multi-versions of a dependency more frequently then its direct use, but this should be resolved by the module system at build time. In the case a developer is using multiple versions of a dep directly, it is an expert-level feature and they would have to be thoughtful in using both versions, and at which point the resolution requires the explicit path via SIV. I also believe that for single major version uses, even without the version suffix on the import path, developers have been trained this way across all other ecosystems, and with code hinting, clicking through, or opening the source-of-truth go.mod to view the version details is how I'd do it in any other language, and its still how I do it in Go. Therefore I believe the mental model when reading source will remain intuitive, and even more readable.

And I don't understand what is supposed to happen if the module is a library imported by a main package that also imports other library packages that themselves import "frobozz.gopher/leftpad".

I understand this is the tricky one. I have a suggestion here too.

As @bcmills is describing the automation tools for updating import paths within an application's source -- I believe automation is a good idea, but what do you think about this? ..instead of applying mass-update automation to application source, as it still creates noise in history/reviews/docs/discoverability/etc., to focus these automation tools in the scope of go.mod and go.sum.

As such, to answer the thought, 'what is supposed to happen if the module is a library imported by a main package that also imports other library packages that themselves import "frobozz.gopher/leftpad"' -- without implying an exact approach for the implementation, imagine that the import path is transformed during compile time by referencing the go.mod dependency graph down the tree from the application/top-level, and for any dependency, applying the same transform to the major version if a SIV path is not used. The idea is it would be as though SIV was included, but included at build-time instead of at the source-level. This does require that the app module graph understand the graph of a dependency as well so that it can infer the major version that is required for the transform, but I believe this data is already available in the go.sum. To loop back to https://github.com/golang/go/issues/32014 which automates updating the source import paths (which is the very issue of the noise), I believe the outcome is the same from a module resolution perspective (but much cleaner from code perspective), as the version is specified in the go.mod/go.sum of the dependency, but optionally included in the import paths (up to the developer).

Just my initial thoughts, I'm also excited to hear from others on possible solutions and happy paths.

[bcmills]

imagine that the import path is transformed during compile time by referencing the go.mod dependency graph down the tree from the application/top-level, and for any dependency, applying the same transform to the major version if a SIV path is not used. The idea is it would be as though SIV was included, but included at build-time instead of at the source-level.

If I understand your suggestion, that would imply that the same import path + identifier could mean different things in different parts of the program. So, for example, if you were looking at a call site for leftpad.Pad in your editor, and ran go doc frobozz.gopher/leftpad.Pad on the command line, you would potentially get the documentation for a completely different version of the Pad function from what you are looking at.

Moreover, that would either cause error messages from the compiler to expand paths to something other than the import path that actually appears in the program source, or reintroduce the confusing cannot use x (type "some/path".T) as type "some/path".T errors that occurred when using vendoring in GOPATH mode. (See #31681, #17822, #30838, and probably many more.)

That is precisely the sort of same-name-different-meaning confusion that modules were designed to address in the first place.