proposal: cmd/go: make major versions optional in import paths

[pkieltyka]

Semantic Import Versioning (SIV) is a novel idea for supporting multiple versions of a package within the same program. To my knowledge and experience, it's the first example of a strategy for supporting multiple versions of a project / dependency in the same application. More importantly though, its clever introductory design allowed it to offer multi-versioned packages in a Go program while maintaining Go's compatibility guarantee.

Multi-versioned packages in a single program can be quite powerful -- for instance, imagine a Web service where you'd like to maintain backwards compatible support for your consumers, you can simply use an older import path from the same origin repository and versioning control, and quite elegantly continue to support those older API versions.

Although SIV may be an elegant solution in the scenario described above, it also adds unnecessary complexity, cost, code noise, discoverability and ergonomics for the majority of packages (publicly and privately) which may not ever have a mutli-version requirement (I'd argue most packages, and simply we can look to other ecosystems to see this is true). I am sure the Go team has heard a lot of feedback on the friction of SIV. https://twitter.com/peterbourgon/status/1236657048714182657?s=21 and https://peter.bourgon.org/blog/2020/09/14/siv-is-unsound.html offers some excellent points as well.

Clearly there is a case for SIV as an elegant solution for supporting multiple versions of a package in a single application, and there is also a strong case to make SIV optional.

It's clear to me there is a design trade-off at hand, and there is no single correct answer. As I consider the 80/20 rule in making an architectural decision between two trade-offs of capability and usability, I prefer to go with the 80% case so long as it doesn't forego the 20% ability. Which design is the best to optimize for if we can still support both? In the case with Go today, its not possible to opt-out of SIV, or opt-into SIV -- I believe both approaches can yield a happy solution. If we were starting from the beginning, I'd suggest to have SIV be opt-in, but maybe at this point its better for it to be an opt-out design to maintain backwards compatibility with history.

---

I'd like to propose a path to make SIV opt-out at the level of an application developer consuming a package, while being backwards compatible with current packages and tools.

I'd like to use https://github.com/go-chi/chi as an example for this proposal which adopted semver ahead of Go modules and SIV, and is built for developer simplicity and ergonomics intended for pro Go developers, but also making it familiar and accessible for developers who are new to Go -- these are my design goals for chi as an author and maintainer as started back in 2017. My present goal is to release Chi v5 without a SIV requirement and the only way I can do so is with the proposal below:

---

Proposal, by example:

github.com/go-chi/chi/go.mod:

module github.com/go-chi/chi/v5

go 1.16

then, git tag chi as v5.0.0 to make the release.

Application developers may consume the package via go get github.com/go-chi/chi@latest or with @v5 or @v5.0.0 and the expected import path will be "github.com/go-chi/chi", however "github.com/go-chi/chi/v5" import path would also be valid and usable.

In the above case, we're specifying the go.mod as expected with current behaviour with SIV from a library perspective. However, from the application perspective when fetching or consuming the library, I may opt-out of the "/v5" suffix in the import path and only adopt it in the scenario when I'd like to support "/v5" and "/v4" (or some other prior version), where I require the handling of multiple versions simultaneously in my program.

I believe the implementation of the above to be backwards compatible as developers would continue to use "github.com/go-chi/chi/v5" with older version of Go as SIV is implied, but optionally developers could make their choice of multiple-version support for the package by handling the import paths themselves and import "github.com/go-chi/chi" to utilize v5.x.x as specified by go.mod.

I believe changes to the Go toolchain for such support would be minimal and would be isolated to the components which build module lists.

Thank you for reading my proposal, and its consideration.

[rogpeppe]

This would clearly raise the possibility of massive confusion when looking at source code.

Yes, there's definitely a possibility of that. I'd hope that the confusion would be no more than the confusion you might see when an import path resolves to different minor versions depending on what module you're looking at.

The current system also has potential for confusion, of course - when a module has many major versions, the distinction between different major version numbers can easily be lost even when included explicitly, especially when imports are largely maintained automatically.

And I don't understand what is supposed to happen if the module is a library imported by a main package that also imports other library packages that themselves import "frobozz.gopher/leftpad".

In my suggestion above, imports are resolved relative to the module that they're inside, so the question "what's the full import path corresponding to this import directive?" can always be resolved by looking at the go.mod file in the module that contains the Go file in question (†).

So it's entirely possible that there would be multiple major versions of a module in the build, but at least there would be a consistent view within a given module.

Two potential advantages of going in this direction:

• make the common case (using a single major version of a dependency) the easiest case
• make it significantly easier to upgrade from one major version to another

The second point in particular is, I think, often hard to appreciate until you've been doing it. When you're upgrading to a new major version, you might need to apply some semantic changes because of the new API. These are the changes that you want to be paying attention to. There are also the syntactic changes - the import path changes - that often dwarf the semantic changes. Sometimes there might be thousands of files changed because of the changed import path, but only one or two changed because of the actual API changes. This presents real challenges:

• github reviews show all files on a single page and won't actually show all the files when there are too many.
• the extensive changes mean that if the PR takes a while to review, it's very common to encounter conflicts, which makes things harder and more error prone as the conflicts are fixed (potentially undoing some of the import path changes).

It's possible that this problem might be amenable to tooling, but I haven't seen a decent proposal in that direction yet.

(†) There are some aspects that would need clarification, for example there would need to be a solid way to distinguish between direct and indirect dependencies in the go.mod file, especially given that the lazy module loading proposal will end up including all modules in the go.mod file.

[pkieltyka]

@bcmills I have to tend to other work atm but I'll circle back with a more thoughtful response on the suggested design and consider your points.

As you've seen this lengthy discussion and prior dialogue on this topic over the years -- do you see no path at which this proposal (or alternative) could be implemented and supported in Go modules to ease the pain in upgrading major versions without an explosion in source between upgrades? or are you saying its effectively impossible to support the high-level goals of this proposal, while maintaining the integrity of the original goals and purpose of Go modules?

[pkieltyka]

In my suggestion above, imports are resolved relative to the module that they're inside, so the question "what's the full import path corresponding to this import directive?" can always be resolved by looking at the go.mod file in the module that contains the Go file in question (†).

yes, this is exactly how I saw it as well and was trying to describe. I understand the caveat of † as well, but this sounds completely solvable and isolated.

[theckman]

I do not understand this confidence. As far as I can tell, the need arises for any module a) that has a diamond in its dependency graph and b) where the bottom of that diamond bumps their major version at some point. I do not understand why you think that would be a small subset of the Go ecosystem. Alternatively, you don't think those criteria are sufficient to imply the need, which seems entirely non-obvious to me as well - personally, I consider the argument Russ makes to be very convincing.

@Merovius in my experience, internal source code at companies depends on external libraries for a few things but much of what's done is using internal code / dependencies. In those environments you have more control over your dependency graph, and can more thoughtfully do version upgrades in a way that make sense. If you hit a case where you need to roman-ride two versions of something, my experience has been that it's always for the short duration of the transition. After the transition is over, I have no need to support two major versions.

If you depend on something that hasn't upgraded versions, it's been more common to see my company try to contribute back to that project so that the upgrade is done... or to fork it and take over maintenance because that project is unmaintained.

If you work at a large company who uses a monorepo for everything, multiple versions + SIV makes sense. If you're working in an environment that consists of one repo per thing, the SIV component not being optional means I am now forced to take on the pain/complexity added so that we can deal with monorepos (and temporary transitions), even though I'm not doing that myself.

Now let's get to the larger meta problem: There's a pattern in the community where more and more module authors are choosing to never release a 1.x version of their module, because of the challenges that SIV presents and that this proposal hopes to mitigate.

To me this seems like we've implemented a technical solution to a social problem, that's encouraging module authors to do something more risky and maybe the opposite of what we want. It also means we're less and less likely to hit the major version diamond dependency problem, because people aren't releasing new major versions. Instead we're going to start having to deal with diamond dependency problems within minor versions, because Gophers are not going to be releasing 1.x versions of their modules.

So what do we do?

Edit: Similar to my request above, if you disagree with something in this post I'd appreciate a comment instead of a :-1: so I can understand what you disagree with and why. It helps influence my own thoughts/opinions.

[mfcochauxlaberge]

Those private repos where SV isn't as important and where all consumers of the repos are known and controlled, can simply stick to v0.x.x, no? Or maybe even just not use semantic versioning... In fact, maybe there's nothing wrong with sticking to v0 even for a public library. You can mention it in the docs that your stuff can be used in production, but there's no commitment to compatibility and a simple commit can break your code. Why not, if that's the level of sophistication you're willing to put in your work. (the last sentence might sound passive-aggressive, but it's not, I actually think it's okay to publish code for free and not take on the responsibility to professionally maintain it as if it was your day job)

This is a big discussion on a proposal that wants to complexify (even if it's just a little bit) a system liked by the community because it's simple and opinionated just for... avoiding three extra characters in the import path? Save the 2 minutes it takes to do a search and replace?

A new major version is a little bit annoying in Go because it's supposed to be. If you keep breaking your library, then this is a problem you need to handle yourself, not impose it on the community.

[peterbourgon]

Those private repos where SV isn't as important and where all consumers of the repos are known and controlled, can simply stick to v0.x.x, no?

I don't think an entire category of Go code should have to opt out of semantic versioning altogether because their rate of change is above some specific and low threshold.

[pkieltyka]

yikes, @bcmills you agree with that statement? it is so clearly wrong

[bcmills]

@theckman

Now let's get to the larger meta problem: There's a pattern in the community where more and more module authors are choosing to never release a 1.x version of their module, because of the challenges that SIV presents and that this proposal hopes to mitigate.

I agree that this seems to be the emerging pattern, but I don't agree that it is intrinsically problematic. If a module author intends to be able to make breaking changes to an established package (at an established import path), then staying at major version 0 seems correct and appropriate — it is a clearer declaration of the intended stability guarantees for the API.

To me this seems like we've implemented a technical solution to a social problem, that's encouraging module authors to do something more risky and maybe the opposite of what we want. It also means we're less and less likely to hit the major version diamond dependency problem, because people aren't releasing new major versions.

That assumes that the consumers of those modules take on dependencies on v0 modules in the same way that they would have taken on dependencies on stable versions. But I don't think that's entirely true — at least at the margin, a user deciding between two modules, or between an external dependency and an internal reimplementation, should be less likely to take on a dependency that may subject them to API churn down the road.

It also means we're less and less likely to hit the major version diamond dependency problem, because people aren't releasing new major versions. Instead we're going to start having to deal with diamond dependency problems within minor versions, because Gophers are not going to be releasing 1.x versions of their modules.

How is that situation any worse than what we end up with if breaking changes are “easy”? If folks are not releasing 1.x versions, and are instead making (and, ideally, contributing downstream fixes for) breaking changes in v0 APIs, then we end up with some amount of downstream churn due to breaking changes. That seems more-or-less manageable.

But if folks are releasing 1.x versions, followed by 2.x and 3.x and so on... don't we have exactly the same problem of API churn? How does moving the number being incremented from the second field of the version to the first field help with that churn?

[DmitriyMV]

So it's entirely possible that there would be multiple major versions of a module in the build, but at least there would be a consistent view within a given module.

@rogpeppe I'm unsure about this in practice. For one thing - transitive exports. Imagine that ModuleA requires ModuleB and exposes ModuleB defined types trough it's own public interface. Now lets say our app requires ModuleA and also requires module ModuleB directly. The usage of ModuleB types across app, and passing them to ModuleA work when they are both of the same major version. But if their versions are different, you get errors about same types not being the same.

You could bypass this, by restricting one "canonical" path to one major version, but that would result in situation where some transitive module down the line doesn't use the same major version as your main app does.

[thepudds]

Hi @rogpeppe , I think I might not be understanding your first bullet here:

Some possible rules:

• select all modules from go.mod that match the repo name.
• if there's more than one, error with "ambiguous import path"
• otherwise select that major version.

... including I'm not 100% sure why the set to examine is coming from the go.mod, and I didn't quite get why lazy loading would impact what you are suggesting, and also I think that text was written in Ye Olden Days when people more frequently informally referred to modules as repos, which might be adding to my mental stumble or parsing error. 😅

Separately, under your proposal, is the requirement that each module must be consistent within the import paths used in the code of that particular module, and the choice made by any given module is insensitive to all choices any other module might make?

One example, suppose both Module A and Module B depend on the same v2+ major version of Module C:

 A -> B -> C              
  \
   \-> C

Is there a scenario under your proposal where the author of Module A starts off using the non-SIV flavor of the import path for C, but then A does an upgrade of B, and A is forced to use the non-SIV flavor in A's import paths for C because B changed its approach? Or any other scenarios outside of this example where just doing an upgrade of one or more dependencies forces an author into one flavor or another?

Those questions are trying to set aside for the moment the impact future lazy loading of modules, but part of what is triggering my questions is wondering why lazy loading would impact your proposal (or at least, it seems it wouldn't need to).

Sorry, just looking to better understand what you proposed, including what is the exact boundary of the consistency requirement.

[shazow]

I wanted to bring up one more related point: API compatibility is a separate problem from functional compatibility.

We can detect API (in)compatibility across versions programmatically even without SemVer, but it's very hard to detect whether the nuances in functionality changed from under the same API--this is what version signaling is really useful for. It's important to be able to signal consumers that "the API may or may not look the same, but the way it behaves has changed so double check that it still works for you before updating."

Sometimes this manifests as security fixes, by removing a footgun scenario that some people may have depended on. This was mentioned earier, and there's some debate about whether security improvements justify breaking API compatibility.

One thing I'm confused about:

We effectively have a way to opt out of SIV: By using 0.x versioning and never bumping the major version.

Maybe people do this because it's more convenient, maybe out of protest of how SIV works in Go, maybe because they're signaling that this project will forever be unstable and you should just deal with it (is this even a useful signal as a consumer of this module? I've used plenty of 0.x dependencies that were far more stable than 1.x dependencies). At minimum, we can agree that there's no way to tell these apart right now.

On the other extreme, as @bcmills pointed out, we have people who don't care about semantic signaling or stability and just bump the major version anytime they're required to by our current SIV policy. Again, we can't tell these maintainers apart from maintainers who are trying to signal functional compatibility.

Hopefully some people in between are using SIV as was intended, but this kind of enforcement is creating a lot of noise on both ends of the spectrum. I feel like this proposal would create a sink for this noise to drain into, and allow maintainers to focus more on signaling functional compatibility while Go tooling can focus on tracking API compatibility for the consumers.

[mfcochauxlaberge]

Those private repos where SV isn't as important and where all consumers of the repos are known and controlled, can simply stick to v0.x.x, no?

I don't think an entire category of Go code should have to opt out of semantic versioning altogether because their rate of change is above some (very low) threshold.

Read question "If even the tiniest backwards incompatible changes to the public API require a major version bump, won’t I end up at version 42.0.0 very rapidly?" on https://semver.org.

A rate of change is above some (very low) threshold might happen, even on a top quality library, but that should be an exception and no need to complexify the Go tool for those edge cases.

yikes, @bcmills you agree with that statement? it is so clearly wrong

@pkieltyka Which statement exactly? Can you elaborate why it's wrong? What @peterbourgon quoted from me was just a question. I found the tone disrespectful. :(

[Merovius]

It seems to me, that if

• The author of module foo could opt-out of SIV, probably by something in go.mod, possibly on a per-dependency basis (note: as SIV is the default, opt-out seems more natural to me, but it could just as well be opt-in)
• If they do, all imports in module foo of that dependency must not mention a version in the import path
• The go tool, for such imports, transparently replaces the import paths with their SIV equivalent, when compiling module foo

[edit: Tried to clarify a bit based on the follow-up comments. See also below for an example]

we would end up

• Preserving the ability to use multiple major versions in a single binary
• Preserve the requirement mentioned by @ianlancetaylor that different packages use different import paths (it effectively implements the solution he outlined)
• Satisfy the goal of this proposal

So, it seems to me, that is a version of this proposal that is most likely to be accepted and satisfy both sides. If that's the case, I think it would be most productive if we assumed that version, for now - and put the question of how vital or detrimental SIV is aside. If there is an option on the table that makes the question immaterial (because you can choose), it seems most productive to discuss the feasibility, advantages and disadvantages of that option.

Personally, I could live with this option. I do agree that we shouldn't make upgrading to a new major version harder than it has to be - I'm fine accepting having to rewrite all imports, if there is a clear benefit, but if we can get most of that benefit without, I don't see why we need to intentionally increase pain (and yes - I do agree that major version bumps should be rare - just that unnecessary toil is the wrong incentive to use).

AFAICT the main objection is, that the transparent rewrite of imports can be confusing in some situations (we mentioned docs and error messages, I believe). And the main open question is, if the rewrite is feasible, i.e. if it gives reasonable results with some dependency graphs (probably not summarizing this well, as I don't understand the pitfalls well enough myself).

Is that a fair summary, so far?

@theckman I did not comment about my 👎 because I disagree with almost everything you said in that comment, but I don't think litigating those disagreements is useful for the discussion or pertinent to the issue. It's getting increasingly off track.

[theckman]

@Merovius The issue with your amendment suggestion, if I understand it correctly, is that it then you cannot roman-ride between two major versions to support a transition for a short period of time, because the author has declared that you should not import them using SIV. The proposal as it's currently written would preserve this powerful ability.

I think making it a per-module configurable option doesn't really address another one of the core problems this proposal tries to solve: the cognitive dissonance that happens when you have to refer to something by a different name when its version changes. By making it a configurable option in the Module, that's going to cause quite a bit of cognitive overhead that each individual must carry as they need to try to reason about the correct way to use a Module and its limitations.

In complex Sociotechnical systems we should be striving to reduce as much cognitive overhead and dissonance as we can, as we know (based on a whole area of research) it's shown to increase the risks of people making decisions with partial context which can in-turn lead to unintended side effects.

[nemith]

Is the onus of SIV on the module createor or the module consumer? Is the reluctance of bumping major versions by module owners due to the burden they are putting on their consumers? It's not clear to me with the proposal who the who benefit the most or which side of the module (owner vs consumer) is affected the most.

In my mind this is being driven from a module owner pov. However, it seems like the actual burden is on module consumer which in turn is being represented as a support issue for the module owner.

Perhaps we should be looking at this as a consumer solution where version imports can imply a version (perhaps via go.mod) with an easy way for module creators to inform how to upgrade/support major bumps? I think a solution where the module create is controlling opt-in/opt-out of SIV would cause a lot more confusion and issues.

[Merovius]

@theckman

The issue with your amendment suggestion, if I understand it correctly, is that it then you cannot roman-ride between two major versions to support a transition for a short period of time, because the author has declared that you should not import them using SIV.

You misunderstood. My intention is that you can write into your go.mod file something like

module mydomain.com/foo

require example.com/bar v2.3.0 nosiv // completely made-up syntax - attached to the individual require directive, to enable you to do this on a per-module basis

If you do that, you can now import "example.com/bar/pkg" and when compiling your module, cmd/go will rewrite this transparently into import "example.com/bar/v2/pkg". Neither does the author of example.com/bar have anything to say about whether or not you are opting out of SIV for their module, nor do people who use your module be in any way influenced by you opting out of SIV for some or all of your dependencies.

You could, if you want to depend on multiple versions, then do something like

module mydomain.com/foo

require example.com/bar v2.3.0 nosiv
require (
    example.com/baz/v2 v2.0.0
    example.com/baz/v3 v3.0.0
)

Given that you are not opting-out of SIV for baz/v2 and baz/v3, you import them via

import (
    baz2 "example.com/baz/v2"
    baz3 "example.com/baz/v3"
)

And use them accordingly.

I am strongly opposed (as I said above) to any version of this proposal that would propagate the effect of opting out of SIV beyond the module that is making the decision. To put it another way: I neither want to be prevented from using your module if you opt out of SIV, nor do I want to receive tickets that I have to do something to allow a reverse dependency of mine to opt out :)

I think making it a per-module configurable option doesn't really address another one of the core problems this proposal tries to solve: the cognitive dissonance that happens when you have to refer to something by a different name when its version changes.

I understand the criticism. Personally, I believe that is a UI issue that should be separately addressed - and that it's detrimental to insist on coupling the two. But, if you can't accept a solution that makes SIV optional if it doesn't also address that criticism, that's your prerogative.

@nemith

Is the onus of SIV on the module createor or the module consumer?

It's hard to answer this question correctly, because a Go programmer is usually both. I hope my example above makes this clear - opting out of SIV should, if anything, only affect the compilation of the module doing it and not require interaction from anyone else. So if I understand the question correctly, I think it should be the consumer.

[theckman]

I understand the criticism. Personally, I believe that is a UI issue that should be separately addressed - and that it's detrimental to insist on coupling the two. But, if you can't accept a solution that makes SIV optional if it doesn't also address that criticism, that's your prerogative.

@Merovius Unfortunately, I think two topics are more intrinsically intertwined when we're talking about how we as humans understand the system, and the components that go into it. The way imports are written and how we get new versions using go get are totally related. How we think about those things, and the decisions we make / actions we take, are going to be informed by that context and the intersection of these two challenges.

I'm much less interested in a solution that introduces more cognitive dissonance / overhead on the humans who make up the system. I feel confident that trying to solve them as distinct issues itself would not recognize the interplay that is actively happening, and will continue to happen.

This interplay between the two is why we're seeing Module authors choose to never release v1, and to instead rely on v0 until the end of time (even if their API is stable). Because of the challenges with getting the Module with the new major version (go get), as well as how to make use of it in code (because of the imports needing updated). Both are contributing to that, and solving only one doesn't address that larger issue and risk to the ecosystem.

If we address both, I believe we will remove the incentives that Modules authors have to never release a v1 version. If we can do that, we're in a much better place as an ecosystem as we can start to depend on versions to have semantic meanings again.

[Merovius]

@theckman Okay. I tried my best to steer the conversation towards something with a reasonable chance of reaching consensus, to address the actual issue this proposal is, AIUI, trying to solve - the need of having to rewrite import paths when upgrading a dependency to a new version. If I've failed in that, then so be it.

Both are contributing to that, and solving only one doesn't address that larger issue and risk to the ecosystem.

I did not suggest to solve only one. Quite the contrary, I explicitly suggested to solve both. Just that not both need to have the same solution. My prediction is that by refusing to accept a solution to either, unless it also solves the other at the same time, you would end up solving neither. But, as I said, whether or not you want to try that out yourself is up to you.

[bcmills]

@shazow

We can detect API (in)compatibility across versions programmatically even without SemVer, but it's very hard to detect whether the nuances in functionality changed from under the same API--this is what version signaling is really useful for. It's important to be able to signal consumers that "the API may or may not look the same, but the way it behaves has changed so double check that it still works for you before updating."

At scale, “double check that it still works for you” is basically impossible, because you have no idea what invariants the other libraries you depend on are themselves depending on. (This is a corollary of Hyrum's Law.)

As @ianlancetaylor noted in http://golang.org/design/28221-go2-transitions#language-redefinitions:

The complexity of a redefinition is, of course, that we can no longer rely on the compiler to detect the problem. When looking at a redefined language construct, the compiler cannot know which meaning is meant.

If the run-time behavior of a function changes but it does not result in a build break, then it is extremely labor-intensive for a maintainer to audit every call site to determine whether that call site requires the new behavior, the old behavior, or (perhaps) is compatible with both. They can't necessarily even rely on when the code was last audited, because the last audit may have missed some call site.

In contrast, if the new behavior is give a new name, then the intent is completely unambiguous: the new identifier has the new behavior, and the old identifier has the old behavior, full stop. There is no breaking change (and no need to bump the major version), because the old identifier continues to have its existing meaning.

[myitcv]

Drive-by comment: one problem with a solution that interprets an import path according to something outside of the source file in which it is contained is that code is much harder to share. Consider some example code on Stackoverflow that references example.com/blah. I can't copy-paste (or share) such code without further context.

[mvdan]

@myitcv makes a good point. It reminds me a bit of how copy-pasting a link to a file on master on a GitHub repo will likely work today, but might break at any point in the future. The only way to make a reasonable "perma-link" is to press y, linking to a commit hash instead of a branch/tag name.

What's tricky there is that many people forget to do that, or perhaps don't even know about it. I imagine the same kind of issue would happen with Go code involving ambiguous imports, where people might forget to replace the example.com/blah import with example.com/blah/v4 unless they don't want to also copy their go.mod.

[veqryn]

go modules should have just forced people to use v0 and v1 in their import paths and module names. Make people deal with it from day 1, then there wouldn't be so much confusion when people try to release or upgrade to a v2 or higher. Go could also warn whenever it detects two versions of the same repo used within the module (not including its dependencies or vendored code), to hopefully alert people who may have missed an import line when upgrading a dependency.

[Helcaraxan]

To add yet another perspective / summary of many of the above made arguments:

• The wide majority of identified issues that would encourage implementing this proposal revolve around the authoring of code.
• The wide majority of issues that the proposal's implementation would create revolve around reading and understanding code.

In the past where trade-offs have been necessary between making code easier to read and understand and making code easier to write Go has almost always favoured the former over the latter because "code is written once but is read many times".

As to the more hands-on arguments I had not taken the time to write them down earlier but @ianlancetaylor and @myitcv have already summarised my key concerns.

Debugging non-trivial bugs that involve diamond-dependencies using different majors could become very confusing. While browsing through my dependency's code via my IDE I might find the same import path in two dependencies but they may correspond to two different majors which are yet different from the major version used by the main module. Add onto that some global state in the underlying package and it becomes dramatically harder to understand what's going on.

[Merovius]

@myitcv

I think that also applies to replace directives, doesn't it? It technically even applies to normal modules - if you use an API defined in v1.2.0, it might not be available in v1.1.0, so without the context of the minor version in go.mod it won't work. And as long as we talk about the content of a single file, even declarations in a different file of the same package affect compilation.

I'm not saying the issue doesn't exist, but like most problems it's a matter of degree. We already accept that some context outside of a file is necessary. I'm not sure it would be getting that much worse.

[myitcv]

I'm not saying the issue doesn't exist, but like most problems it's a matter of degree

Fully agree.

I'm not sure it would be getting that much worse.

This is where I would tend to disagree.

Yes, replace directives are an exception (but they should not form the basis of examples/documentation, generally speaking), and additive changes within a major version >= 1 might be somewhat surprising, but I don't think this justifies/supports adding another category of situations where that extra context is required. Put another way, there a higher probability that a new user will run into an issue under such a proposal.

More broadly, I'm much more of a proponent of improving rf and associated tooling to automate major version upgrades as far as possible. Such tooling helps to alleviate the one-off pain of such upgrades, review etc.

[jaloren]

My thoughts are similar to @Helcaraxan.

I work heavily in both go and java communities -- specifically spring. I have seen what happens when a community does not follow semver, regularly introduces breaking changes in minor and patch versions, and refuses to provide a unique name each time a package has a breaking change and you actually need both packages. Classpath loading hell and class shadowing are some consequences of the community refusing to change package namespaces when a breaking change is introduced. In short, I found Russ Cox's arguments compelling and I really appreciate the design that came from it.

As a consumer of external third party libraries and someone who cares a lot about software reliability and compatibility (if something breaks, lots of money will be lost)

• I highly value the fact that if I depend on a module and it has a breaking change, I must opt into that software update by adopting a different namespace
• I highly value that if there's breaking change no matter how small it forces me to actually change the name of the package because its clearly communicating that 1) breaking change has occurred and 2) and that I am modifying my code to fix this software incompatibility. It means that its explicit instead of upgrading, having things break silently, and finding that out the hard way when the software is deployed to prod.
• I highly value knowing that a library author makes breaking changes but intentionally chooses to not bump the major version because they don't want to force the consumer to spend a minute running a sed script to update import paths. That means I know what libraries not to use.

[yiyus]

All proposals here have been to solve the issue at the tool level, but I think most practical problems would disappear if it was solved at the language level instead.

Imagine that we could define package aliases using the package keyword (there may be better ways to do this, I am not proposing this syntax, it is just an example). Then:

package mypkg
package blah = "example.com/blah/v4"
import blah

And, from any other file in the same package:

package mypkg
import blah

We still have the extra level of indirection (with all its advantages and disadvantages), but we do not need the extra context in the go.mod file. Everything is in the source files. The problems of the user with import blah would not be more than with a variable or a function defined in another file of the same package, but the developer would only need to update each import path in a single place (going a bit further, all imports may be centralized, by convention, in an imports.go file, for example).

I think such an approach would solve the issues raised here while still making obvious where each package comes from (obvious, but of course with one extra level of indirection, with its inconveniences). The main question is if this extra level of indirection, explicitly avoided so far, is worth it now.

[Merovius]

@jaloren Some of what you are saying seems to imply that under this proposal, you could unknowingly switch major versions. I just want to be clear that that's not the case and it's certainly no ones intention. You would still need to explicitly bump the version of a dependency you are upgrading. The only change would be, that you only have to do that in go.mod.

I also can appreciate that you like having the major version in the import path, as an extra signal. I tend to agree. But, to be clear again, no one intends to take that away from us. If this proposal was adopted in a form like I outlined above, you can still have all the advantages you mention, by simply not using the nosiv marker (or whatever that ends up being). That is, under this design, if you do nothing, nothing changes for you. It will also probably something you can relatively simply grep for, to enforce it as a standard in your organization.

I just don't want there to be any misunderstanding of what the intent is (at least as I understand it). It's to give people who don't feel that SIV is worth the benefit an escape hatch - without affecting anyone but themselves. It's not to get rid of SIV altogether.

Of course, if you consider "it disincentivizes people from releasing a v2+" a benefit of SIV, that benefit would likely go away and that certainly needs to be considered when deciding on this proposal :)

[jaloren]

@Merovius There seemed to be multiple proposals floating around here and a variety of comments hostile to the current design of SIV so I wanted to make sure that people who find this valuable like myself are heard when considering cost and benefits of the various options.

I did see your proposal and my comments were not directed at it. Your proposal sounds quite promising and I think something like that would be fine as long as its opt out and the consumer is the one that gets to make that choice. In other words, I think its a compromise that could work for both camps (though I can only speak for my camp of one :) )

[bcmills]

@Merovius

if you use an API defined in v1.2.0, it might not be available in v1.1.0, so without the context of the minor version in go.mod it won't work.

The key difference there, I think, is that the version added by go get or go mod tidy by default is the latest version of the module. So if the module's API is actually stable, code written against v1.1.0 should still work when built against v1.2.0.

That is: with semantic import paths, if you just copy the code from whatever document you are reading and run go mod tidy as normal, then as long as the module's API is stable as advertised that code will still work unmodified.

[bcmills]

@yiyus, I don't see how putting the indirection in a different source file in the same package would be fundamentally better than putting the indirection in the go.mod file. Either way, the change in meaning requires the reader to notice information that is not local to the current file.

[pkieltyka]

@jaloren I agree it's crucial to offer a comfortable semver experience for all developers, whether from the perspective of a library author or application developer.

However, this proposal is looking to assist with that very goal instead of retract from it. That is: to make semver a better experience for everyone, while maintaining backwards compatibility with Go tools, SIV and Go modules.

As @theckman has clearly stated https://github.com/golang/go/issues/44550#issuecomment-785259915 -- "There's a pattern in the community where more and more module authors are choosing to never release a 1.x version of their module, because of the challenges that SIV presents and that this proposal hopes to mitigate.". @jaloren as you can imagine, I'm aligned with you, but this problem is very real -- consider the avoidance of v1/v2+ in Go a kind of "schelling point" in our community, of which is not controllable, happening organically, and is important to address. Failing to address this problem in my opinion will hurt Go's developer experience forever. And anyone who disagrees can just look at the size of this thread over the last 48 hours as one data point, and then read the passionate words from many.

I am one of those module authors who never wants to release a 1.x version of a package to avoid the code noise required by major version bumps with SIV, and the very reason I submitted this proposal. As I attempted to state in the proposal text, I understand SIV's benefits and agree they're important, but I've also proposed a small backwards-compatible iteration to smooth out the developer experience from the perspective of a library consumer -- notably my suggestion is optional hence the name, as I'm effectively proposing an alias for the highest major version of an import path, validated by go.mod at build-time.

In case it's been missed, I did create some basic diagrams: https://github.com/golang/go/issues/44550#issuecomment-785112275 -- you will notice these diagrams are identical to if you took a cross-sectional view of modules A, B, C, D from Go 1.15 or 1.16 today. The ideas in this proposal are in the blue text. This is because as I stated, my goal is to keep things backwards compatible (maybe there would be a very minor worthwhile change as it relates to v0 and v1 and @latest, but I'm not fully sure yet if that is needed or not).

Perhaps the name of the proposal "optional SIV" or to say "opt-out" aren't the right words and are a bit confusing, but those are the best way I could describe at the time. I do believe @rogpeppe did a much better job than me.. as he said: https://github.com/golang/go/issues/44550#issuecomment-785051499 and https://github.com/golang/go/issues/44550#issuecomment-785227107

To re-iterate Roger's words, and to state the approach to where I think is the biggest mental barrier, please consider:

imports are resolved relative to the module that they're inside, so the question "what's the full import path corresponding to this import directive?" can always be resolved by looking at the go.mod file in the module that contains the Go file in question

So it's entirely possible that there would be multiple major versions of a module in the build, but at least there would be a consistent view within a given module.

For anyone interested to help solve the problem at hand, please see https://www.figma.com/file/6FxqJsnSBty704OKf4C02b/Go-proposal-44550?node-id=13%3A2 and offer feedback -- I think we can get to the bottom if this is possible or not with some technical diagrams and talking about specific points. Other proposals / ideas are welcome of course, and thank you in advance.

[jaloren]

@pkieltyka This thread is quite long so I may have missed but what did you think of @Merovius proposal?

[ianlancetaylor]

As @theckman has clearly stated #44550 (comment) -- "There's a pattern in the community where more and more module authors are choosing to never release a 1.x version of their module, because of the challenges that SIV presents and that this proposal hopes to mitigate.". ... I am one of those module authors who never wants to release a 1.x version of a package to avoid the code noise required by major version bumps with SIV, and the very reason I submitted this proposal.

I don't understand why this is considered to be a bad thing. To me it seems exactly right. I myself have a couple of packages for which I don't plan to add a 1.x version. I don't want to have to support old versions and I don't want to have to provide strict backward compatibility. I want people to always use HEAD. So I don't use a semver version. As far as I am concerned that is working as intended. I am intentionally pushing support issues off to other people who care.

[pkieltyka]

@jaloren sorry to make more noise in this thread -- but can you link to which comments comprise his proposal and ill happily offer my opinion

[bcmills]

@pkieltyka

Failing to address this problem [the avoidance of v1/v2+ in Go] in my opinion will hurt Go's developer experience forever. And anyone who disagrees can just look at the size of this thread over the last 48 hours as one data point, and then read the passionate words from many.

In response to a similar comment from @theckman, I asked (in https://github.com/golang/go/issues/44550#issuecomment-785310548):

[I]f folks are releasing 1.x versions, followed by 2.x and 3.x and so on... don't we have exactly the same problem of API churn? How does moving the number being incremented from the second field of the version to the first field help with that churn?

Others on this thread have made similar points: it is not obvious to us why “avoidance of v1” is intrinsically a problem — or, really, why it is any worse than releasing a v1 and then subsequently releasing one or more incompatible major-versions.

To move the conversation on this point, we need more than just “passionate words”. What are the concrete harms of sticking to v0, especially as opposed to releasing a series of multiple “stable” versions?

[jaloren]

@pkieltyka here you go https://github.com/golang/go/issues/44550#issuecomment-785333656

[pkieltyka]

@ianlancetaylor

I don't understand why this is considered to be a bad thing. To me it seems exactly right. I myself have a couple of packages for which I don't plan to add a 1.x version. I don't want to have to support old versions and I don't want to have to provide strict backward compatibility. I want people to always use HEAD. So I don't use a semver version. As far as I am concerned that is working as intended. I am intentionally pushing support issues off to other people who care.

That user-story makes sense to me and is completely valid. However, I have a similar story where I want to offer a similar experience, but I'd like to use semver. I believe there are many like me.

[pkieltyka]

@bcmills @ianlancetaylor I also want to mention that I acknowledge the Go team is better to make the decisions for the project than I and my trust is in your collective leadership, as after all it is your team who has gotten us to Go in the first place. I recognize my blind spots. I'm simply trying to offer my perspective and taste, perhaps too passionately at times -- but I do feel there is pain, and others feel it too, and I acknowledge there are design tradeoffs and ultimately there will need to be a stance (ie. status quo, or a change, or alternate) -- and I sincerely accept and thank you for Go's leadership.

The proposal text is something I've wanted to bring up for a long time, and since my issues with chi for years related to Go modules, I felt it would be missed if I never even made the suggestion I've been thinking about for sometime in an effort to circulate some ideas, feedback, or progress. I sincerely appreciate everyone even giving the idea any time and thank you for the openness. I trust the outcome (even no change) will be the right one.

[ianlancetaylor]

Thanks!

[yiyus]

@bcmills The difference is that, although the information is indeed not local to the source file, it is local to the package source code. We have lots of important information that is not local to the source file (variables, types, functions, almost everything can be defined in another file) and it has not been a problem. However, I would see a problem if, for example, type aliases were defined in go.mod files instead of in .go files, and I do not see why package aliases should be different.

I agree that, in practice, the user experience would be quite similar, but I just find it more natural (I admit this is an opinion and not an objective fact). If we can give a different name to a constant in go source files, why should we use go.mod to give a different name to an import path? Also, I think that adding the aliases in the mod file may be problematic for some tools. I expect go tools to parse my go files. However, I am not sure they (or at least not all of them) should care about go.mod. Now, the import path can be resolved just looking at go files (with the exception of replace directives which I, maybe wrongly, consider exceptional), and the go.mod file is only used for reproducible builds. I think that if package aliases were allowed (which I still doubt will be), it would be desirable to keep this property.

[bcmills]

@yiyus: hmm, I think I see what you're saying now!

The declaration

package blah = "example.com/blah/v4"

would introduce a new identifier blah, which would be local to (and unambiguous within) the package.

And because the identifier is local to the package — and, importantly, looks like a local identifer — it can't possibly be confused with anything in the “global” import-path namespace (such as a reference in another package to the related-but-different path"example.com/blah").

I agree that that at least seems a lot cleaner than allowing the meaning of a global import path to vary from one location to another.

[theckman]

I don't understand why this is considered to be a bad thing. To me it seems exactly right. I myself have a couple of packages for which I don't plan to add a 1.x version. I don't want to have to support old versions and I don't want to have to provide strict backward compatibility. I want people to always use HEAD. So I don't use a semver version. As far as I am concerned that is working as intended. I am intentionally pushing support issues off to other people who care.

@ianlancetaylor that's a completely different thing, and absolutely a valid case we should (and do ) support. The issue I am describing is different, and is when a Module author wants to release stable versions that their consumers depend on, where they use semantic versioning as a social contact with their consumers. This is a social contract our tooling understands.

The issue today is that we are encouraging these folks to never release a v1 because of how Modules work. They aren't doing this to push the support burden off on others like you indicate you would be doing, but instead to try and avoid it falling on their shoulders because of SIV. While those may seem like the same thing at the surface, the intentionality of who owns the burden is different.

I don't think avoiding API stability is the impact we wanted to have on our ecosystem by releasing Modules as-is, and if it is... what do we expect the long-term outcome to be? Or maybe more directly asked, why is that what we want?

[dylan-bourque]

To provide an alternative perspective on this, at CrowdStrike we have ~110 modules today and will eventually have 400+ once all of our internal libraries and services have been updated.

We are pushing frontiers on several fronts as part of our normal innovation and to stay ahead of malicious actors and our competitors. This inevitably means that there will be breaking changes in our modules' APIs. The (intentional) extra effort required by SIV is a real logistics problem for us. Staying at v0.x isn't really feasible because our Release Management and Auditing teams have a "no pre-release code in production" stance, which is very reasonable, and tagging v1 is the accepted/standard indicator that a thing is "stable".

With 100s of projects to update, bumping major versions is a nightmare even if it is scriptable. "Just search-and-replace" doesn't sound so great when it has to happen 100s of times across dozens of teams and almost all of the changes in almost all of those PRs would be appending /vN to import paths.

With the v2 discovery gap (i.e. go get example.com/foo will never pull in foo@v2.x.y), every developer we have that has needed to add a reference to either of the 2 v2.x modules we have has run into problems. They add "example.com/foo" to their import block but the build fails. One of us that is already familiar with the issue has to explain (again 😩 ) that, only for those 2 modules, you have to remember to use "example.com/foo/v2" as the import path.

For our R&D processes, we want @latest to mean "the most recent release" regardless of major version, especially for our internal modules. Both of our v2 modules were bumped because v1 was "broken" and we don't want anyone using it anymore. To account for this internal requirement, we've even started looking at writing custom tooling that can check for the existence of a vNext major version of an existing v0/1 module dependency to streamline the process. In the meantime, though, we're essentially advising teams to stay on v1.x, even for breaking API changes. 😞

As far as this proposal is concerned, we would definitely be interested in a path that would allow for not appending the major version to the end of every import. I have lots of other opinions about the pros and cons of SIV as well but they're not really germane to this proposal.

[theckman]

I feel like we would have a much easier time aligning on our concerns with a conversation via a VC instead of an async back and forth on GH. @ianlancetaylor @bcmills @pkieltyka @peterbourgon @lyoness @dylan-bourque (and anyone else following) is that something that we can do?

[peterbourgon]

@ianlancetaylor

I myself have a couple of packages for which I don't plan to add a 1.x version. I don't want to have to support old versions and I don't want to have to provide strict backward compatibility. I want people to always use HEAD. So I don't use a semver version.

If you consider those packages truly unstable, then this is of course your call. But semver v1+ does not require that old versions are supported, or that strict backwards compatibility is maintained (across major versions), or that you as an author intend to support anything other than HEAD. It merely provides a protocol for describing the impact of changes, nothing more. You shouldn't [need to] opt out of those important benefits because of the ancillary concerns you're describing. They're immaterial.

[ianlancetaylor]

Semver requires paying attention to backward compatibility to know when to update to a new major version.

[jaloren]

@dylan-bourque would @Merovius proposal https://github.com/golang/go/issues/44550#issuecomment-785333656 address the challenges you have with module upgrades?

[dylan-bourque]

@jaloren Yes, to a large extent. There's still the UX gap of discovering that a new version of the module with an incremented major version exists, but that's kind of a separate issues (that I think is already covered in other GH issues).

[peterbourgon]

@bcmills

What are the concrete harms of sticking to v0, especially as opposed to releasing a series of multiple “stable” versions?

Sticking with v0 prevents authors from signaling the semantic impact of a version bump, which is a primary purpose of semantic versioning. This can have significant downstream consequences, like the friction created with ops teams and/or organization policy described by @dylan-bourque, or failures in tools that parse and act on semantic versions.

Releasing a series of multiple stable versions is using semantic versioning as it is intended, but, due to SIV, it results in significant incidental toil for both authors and consumers. Additionally, SIV's unique position that there is essentially no relationship between different major versions of the same module, and the downstream effects of that position on tooling, causes significant confusion and toil for consumers, especially during discovery.