proposal: cmd/go: make major versions optional in import paths

[pkieltyka]

Semantic Import Versioning (SIV) is a novel idea for supporting multiple versions of a package within the same program. To my knowledge and experience, it's the first example of a strategy for supporting multiple versions of a project / dependency in the same application. More importantly though, its clever introductory design allowed it to offer multi-versioned packages in a Go program while maintaining Go's compatibility guarantee.

Multi-versioned packages in a single program can be quite powerful -- for instance, imagine a Web service where you'd like to maintain backwards compatible support for your consumers, you can simply use an older import path from the same origin repository and versioning control, and quite elegantly continue to support those older API versions.

Although SIV may be an elegant solution in the scenario described above, it also adds unnecessary complexity, cost, code noise, discoverability and ergonomics for the majority of packages (publicly and privately) which may not ever have a mutli-version requirement (I'd argue most packages, and simply we can look to other ecosystems to see this is true). I am sure the Go team has heard a lot of feedback on the friction of SIV. https://twitter.com/peterbourgon/status/1236657048714182657?s=21 and https://peter.bourgon.org/blog/2020/09/14/siv-is-unsound.html offers some excellent points as well.

Clearly there is a case for SIV as an elegant solution for supporting multiple versions of a package in a single application, and there is also a strong case to make SIV optional.

It's clear to me there is a design trade-off at hand, and there is no single correct answer. As I consider the 80/20 rule in making an architectural decision between two trade-offs of capability and usability, I prefer to go with the 80% case so long as it doesn't forego the 20% ability. Which design is the best to optimize for if we can still support both? In the case with Go today, its not possible to opt-out of SIV, or opt-into SIV -- I believe both approaches can yield a happy solution. If we were starting from the beginning, I'd suggest to have SIV be opt-in, but maybe at this point its better for it to be an opt-out design to maintain backwards compatibility with history.

---

I'd like to propose a path to make SIV opt-out at the level of an application developer consuming a package, while being backwards compatible with current packages and tools.

I'd like to use https://github.com/go-chi/chi as an example for this proposal which adopted semver ahead of Go modules and SIV, and is built for developer simplicity and ergonomics intended for pro Go developers, but also making it familiar and accessible for developers who are new to Go -- these are my design goals for chi as an author and maintainer as started back in 2017. My present goal is to release Chi v5 without a SIV requirement and the only way I can do so is with the proposal below:

---

Proposal, by example:

github.com/go-chi/chi/go.mod:

module github.com/go-chi/chi/v5

go 1.16

then, git tag chi as v5.0.0 to make the release.

Application developers may consume the package via go get github.com/go-chi/chi@latest or with @v5 or @v5.0.0 and the expected import path will be "github.com/go-chi/chi", however "github.com/go-chi/chi/v5" import path would also be valid and usable.

In the above case, we're specifying the go.mod as expected with current behaviour with SIV from a library perspective. However, from the application perspective when fetching or consuming the library, I may opt-out of the "/v5" suffix in the import path and only adopt it in the scenario when I'd like to support "/v5" and "/v4" (or some other prior version), where I require the handling of multiple versions simultaneously in my program.

I believe the implementation of the above to be backwards compatible as developers would continue to use "github.com/go-chi/chi/v5" with older version of Go as SIV is implied, but optionally developers could make their choice of multiple-version support for the package by handling the import paths themselves and import "github.com/go-chi/chi" to utilize v5.x.x as specified by go.mod.

I believe changes to the Go toolchain for such support would be minimal and would be isolated to the components which build module lists.

Thank you for reading my proposal, and its consideration.

[beoran]

@dylan-bourque , As @bcmills pointedout higher in the thread, there are tools (in the making) which will help you to automatically upgrade from v1 to v2 and beyond. For people who are able to use semver, what we need is better tools, preferably integrated with go mod.

However, I think the core problem here is that semver is not always applicable, and that there are situations where it can't be used. Furthermore there is some debate about semver (e.g. https://gist.github.com/jashkenas/cbd2b088e20279ae2c8e), and "romantic versioning" has been proposed as an alternative (https://github.com/romversioning/romver). So the core question seems to be: do we have to change the way go module work in order to allow other versioning schemes than semver?

[peterbourgon]

@Merovius

My intention is that you can write into your go.mod file something like require example.com/bar v2.3.0 nosiv . . . import "example.com/bar/pkg" [in your Go source files] and when compiling your module, cmd/go will rewrite this transparently into import "example.com/bar/v2/pkg".

Combined with changes to tooling so that the identifier example.com/bar is universally understood as versionless, rather than at major version 0 or 1, this suggestion is basically what I would hope for in a solution 👍

Observing that the need to include two major versions of the same dependency in a single compilation unit is the exception rather than the rule, nosiv should also probably be the default, and siv the opt-in alternative.

[theckman]

Observing that the need to include two major versions of the same dependency in a single compilation unit is the exception rather than the rule, nosiv should also probably be the default, and siv the opt-in alternative.

I probably should noodle on it a little more, but I think that sounds ideal.

[peterbourgon]

Drive-by comment: one problem with a solution that interprets an import path according to something outside of the source file in which it is contained is that code is much harder to share. Consider some example code on Stackoverflow that references example.com/blah. I can't copy-paste (or share) such code without further context.

Essentially every other modern programming language identifies dependencies in source files without any mention of version. I don't think this is a meaningful problem.

[Merovius]

Combined with changes to tooling so that the identifier example.com/bar is universally understood as versionless, rather than at major version 0 or 1, this suggestion is basically what I would hope for in a solution 👍

To clarify: If not combined with that immediately (i.e. if that was discussed separately), would that still be a 👍, or would it become a 👎?

Observing that the need to include two major versions of the same dependency in a single compilation unit is the exception rather than the rule, nosiv should also probably be the default, and siv the opt-in alternative.

If we purely did this, current users of modules would break, because any code currently importing example.com/foo/v2/baz (or the like) would become invalid. Note that I used "must not mention a version specifier". So, at the very least, we'd need either

1. Allow both versioned and unversioned import paths. Or
2. A way to opt-in into the "nosiv by default" semantics, e.g. a global directive in go.mod (not attached to a specific require directive).

IMO, 1. is confusing and it might lead to ambiguities (I'm not sure - but I am sure that disallowing both version of import path in a single module rules out such ambiguities). And IMO 2. isn't much better than individual opt-ins and I would assume that you still wouldn't like any option that makes SIV opt-in anyways.

To be clear, doing either is possible. Just explaining the rationale behind choosing the semantics I outlined.

[peterbourgon]

To clarify: If not combined with that immediately (i.e. if that was discussed separately), would that still be a 👍, or would it become a 👎?

If one "mode" of a Go module treats the unversioned import path of a dependency as versionless in import statements, I think it would be enormously confusing if it were not also treated as versionless everywhere else it appears. So I think both are required, but I've no strong opinion on timing or ordering.

If we purely did this, current users of modules would break, because any code currently importing example.com/foo/v2/baz (or the like) would become invalid.

Hmm, is the rationale for that explained somewhere? (I missed it, sorry!) Intuitively, I wouldn't expect nosiv to forbid SIV-style import declarations, I would just expect it to change how the unversioned import path is interpreted. So, your option 1.

[Merovius]

I gave my rationale in the last comment - I would find it confusing and I can't rule out that it causes ambiguities. The first is a subjective judgement. The second is a technical argument, albeit a weak one as it stands - until someone can come up with a concrete ambiguity, I wouldn't object to putting it aside.

I just made that choice because it seemed likely to steer the conversation towards consensus - it provides the wish posed by this proposal (SIV is optional) and it seems to make it obvious that no (or the fewest) technical problems or other negative effects to bystanders are caused.

[icholy]

SIV is a step forward in code clarity. Like fully qualified import paths, it tells me more about the dependency without having to look at go.mod. The only real problem that needs to be addressed (via tooling) is the toil.

[dylan-bourque]

@icholy There are definitely varying opinions about whether or not the version of a thing should be considered a part of its canonical name (as evidenced by multiple long debates on various platforms) but, again, those points aren't germane to this proposal.

Personally, I think SIV creates as many issues as it solves and would definitely like an option to not be forced into it.

[icholy]

@dylan-bourque re: https://github.com/golang/go/issues/44550#issuecomment-786100707

I don't see anything in your comment that better tooling can't fix.

[dylan-bourque]

Well that comment is definitely not the full extent of my thoughts on SIV, only the parts that are directly pertinent to this proposal.

[bcmills]

@dylan-bourque

Staying at v0.x isn't really feasible because our Release Management and Auditing teams have a "no pre-release code in production" stance, which is very reasonable, and tagging v1 is the accepted/standard indicator that a thing is "stable".

That seems like a social problem rather than a technical one. If the release management teams say “this is pre-release code” and the development teams say “this is released code with an unstable API”, then perhaps the developers ought to talk to the release management team to figure out a way to indicate fitness for production use in a way that is more orthogonal to API stability..?

[bcmills]

With the v2 discovery gap … every developer we have that has needed to add a reference to either of the 2 v2.x modules we have has run into problems.

I can understand running into problems by accidentally adding dependencies on APIs that should no longer be used. However, we already have several features implemented — and others in progress — that should help to address that problem more directly:

• retract (#24031, released in Go 1.16)
• per-identifier deprecation (http://golang.org/wiki/Deprecated, implemented in staticcheck in 2017 as SA1019 and likely supported by other tools as well)
• module deprecation (#40357, proposal approved and in progress)
• goimports heuristics for major versions (#41800, under investigation and partially implemented in CL 261084)

Both of our v2 modules were bumped because v1 was "broken" and we don't want anyone using it anymore.

That in particular sounds like an excellent use-case for the Go 1.16 retract directive.

[bcmills]

@peterbourgon

This can have significant downstream consequences, like … failures in tools that parse and act on semantic versions.

Could you give some specific examples of tools whose behaviors could be improved by this proposal?

[bcmills]

Sticking with v0 prevents authors from signaling the semantic impact of a version bump, which is a primary purpose of semantic versioning.

Bumping the semantic version tells the user “something changed somewhere”, but doesn't tell them what or whether they are affected. At scale, that isn't particularly useful for users to know when or whether they should adopt that version, or in what way they would need to change their code to do so — they still need to read the changelog to figure out how they are affected.

In contrast, if an identifier is marked // Deprecated: (in a stable API), or removed entirely or has is signature changed incompatibly (in an unstable API), those specific changes can be surfaced by static checkers and/or the compiler itself based on the user's actual code. Those tools already exist, and — tellingly! — do not depend on the major version of the API as input.

That is also why the Go 2 Transitions document so strongly prefers additions and removals over redefinitions.

---

Automated tools that attempt to upgrade dependencies should already try to compile the code and run its tests with the upgraded version, before committing the upgrades and/or sending PRs for them. If there is a breaking change, and that breaking change is made in a responsible way, then those tools can detect the breaking change even if it occurs in a v0 module.

Moreover, by compiling the code and running the tests, tools can also detect failures due to “semantically non-breaking” changes, such as accidental reliance on undocumented or documented-as-unstable properties of the API.

So I think that the “signaling” lost by staying on v0 is not particularly useful to begin with: it has both too high a rate of false-positives (you bumped the major version, but the breaking change only affects a function that ~nobody uses), and also too high a rate of false-negatives (you didn't make an “incompatible” change but the users' program broke anyway due to Hyrum's Law).

[beoran]

@bcmills Then another "missing" tool would be an integrated API and ABI stability checker like this one for C: https://lvc.github.io/abi-compliance-checker/. There already is go/src/cmd/api but that is not general use.

[icholy]

@beoran There's https://pkg.go.dev/golang.org/x/exp/cmd/gorelease

[ncruces]

So if compatibility is such a huge factor for software success, why want to make it easy to make incompatible changes?

Because, some people, before it was even a rule that major versions should only ever be created if you break compatibility, maybe years ago, made the now unfortunate decision to tag v2.0.0 to signal major feature releases, and now are forced to change the module import path, although they might actually be backwards compatible API wise since even before their v1.0.0.

Yes semver has been a thing for a very long time, but the costs of changing major version while keeping compatibility had not been known. The linking between repo tags and import paths has not been a rule for most of Go's history and it seems to me that, now, it is Go that is breaking something that used to work, and making it particularly painful for these projects to move forward and adapt to modules.

And although I agree that SIV makes perfect sense, at least some consideration should be given to decisions made before it was even a thing, IMO.

[DmitriyMV]

FWIW while I don't think that Go should have a different algorithm for version selection and definition, I do think that some sort of "module restricted aliasing for imported modules" can be useful, just like you can alias imports used by the package. Such import will have a proper full path and name in resulting binary. It would just be easier for developers to alias imports "in module scope" in go.mod, rather than aliasing them per package. To differentiate them from "simple imports", we can import them without quoting marks perhaps? On contrary to the replace such aliases would work on "per module" basis, rather than "global, but only defined in main module".

[Merovius]

@DmitriyMV I think we need to keep in mind that the language is unaware of modules and we probably want to keep it that way, to continue to work with third party build tools like bazel. There might be ways to solve this somehow, but in general, we are treading on dangerous terrain.

[pkieltyka]

@Merovius btw, I think your proposal in https://github.com/golang/go/issues/44550#issuecomment-785333656 is cool too. How do you see it working with existing packages using SIV that want to opt-out, but still be compatible with older versions of Go? (Is it possible?)

[Merovius]

@pkieltyka I don't think so. I don't think we can modify the semantics under which an older version of the go toolchain interprets a program. I think go1.16 would fail to compile a Go module written with an "example.com/foo/bar" import and an "example.com/foo" v2.0.0 directive in go.mod, due to a violation of SIV. Which is, maybe surprisingly, a good thing for the design - a compilation error can be detected in the presence of a go 1.17 (or whatever) directive and warn that a newer Go version is needed. In that sense, this design can be treated as an addition, instead of a redefinition. So a guaranteed compilation failure is at least better than a potential compilation failure. But it does mean that if you want to opt out of SIV, you can only support Go versions which include that feature.

I don't know if it could be made possible to both make SIV optional and be usable with Go versions which don't have that feature. It seems inherently impossible to me, but I don't want to rule out that someone can come up with an obvious idea I've been oblivious to.

That's IMO not a reason not to do it, though. If we think SIV should be optional, it would still be better to start that off now, so it enters the "any reasonably old Go version is supported" window sooner.

(BTW: I want to make clear that, as far as I'm concerned, I was really just trying to take the proposal from the top-post and put it into concrete terms that answer some of the questions I asked :) My personal contribution was to make a couple of choices that IMO make it more broadly acceptable. But I can't fully take credit for it and wouldn't really call it "my" proposal)

[flibustenet]

@ncruces changing the import path on breaking change was always recommended before modules. It was in the faq and we were using https://gopkg.in for example.

I'm not agree about the 80/20% of use case and the small private vs big code. As a solo developer or in small teams we've less infrastructure, scaling, even if relatively small, arrives very quickly. It's why i've always changed the import path on breaking changes, even in other langage since decades. The pain is very small, only few lines of codes (one file for chi) and without risk. On the other side a silent breaking change can be very annoying. It's a very big problem in others langages, it's a killer dead simple feature in Go.

That said, the tools could help us to see when there is new major version and rewrite the import paths on demand...

[ncruces]

@flibustenet

@ncruces changing the import path on breaking change was always recommended before modules. It was in the faq and we were using https://gopkg.in for example.

I'm not sure about always there but that's not what I'm arguing either. What I'm saying is just the opposite, actually.

What if you've kept strict compatibility since v1.0 (because you value compatibility), but tagged v2.0 for marketing or something? You now have to change import paths even though you're compatible if you want to adopt modules going forward.

Yes, in 2020+, working with Go, you know that was a bad idea, that you should only ever change the major if you break compatibility.

But even going by semver, I don't know of any other community that penalizes updating the major this much. So, other communities often do update majors even if they don't break compatibility to signal major changes that (e.g.) might be risky (or just purely for marketing, rather than engineering, purposes).

Go is unique here, and although I don't disagree with SIV, I think some thought/consideration should be given to those who disagree with it, and/or might've make the wrong call before it was even a thing, even if in the end, the decision is to stay the course.

[ernado]

Once I've tried to bump v1 to v2 in one of my projects and it was so painful to update imports everywhere so I've just decided to roll back.

Now I'm just using v0 for any new projects and waiting for SIV to be less painful.

[Merovius]

FTR, I don't think any of these statements are in dispute:

1. Upgrading a major version is never free, in any language
2. SIV increases that cost
3. Increasing that cost means it will (likely) happen less

Personally, I don't really see the point in arguing about these further. We are all in agreement about them.

I feel that arguing about questions we are all in agreement on detracts from the argument what we can do to maybe alleviate the pain.

(Whether or not point 3 above is a good thing is arguably more relevant, because it at least affects whether or not we want to alleviate the pain. Personally, I'm pessimistic we would ever reach agreement on that, no matter what arguments are made. And I believe it's probably more productive to accept that some people feel one way while others feel the other and go from there)

[beoran]

@Merovius Your points are correct, and I agree that we should alleviate the pain. Historically speaking, go has had the tendency to fix similar probelms not by changing the language but by improving the tooling. That's why I think we should ask that the Go developers should integrate the tools mentioned by @bcmills and @icholy into the Go compiler proper. Once we can up a version to v2 easily and get automatic refactoring for this, that would go a long way to solve this issue.

[pkieltyka]

@beoran thanks for your comment

Once we can up a version to v2 easily and get automatic refactoring for this, that would go a long way to solve this issue.

However, since I'm the one who opened this issue, I'd like to offer my response and say that automation tools for doing a search+replace to auto-refactor my source will do almost nothing in helping towards solving for the developer experience goals raised by this issue.

go has had the tendency to fix similar probelms not by changing the language but by improving the tooling

I fully agree, and luckily the Go language doesn't require any changes, but we do need to collectively improve Go modules even if that means a breaking change to the go mod and go get commands. The Go "compatibility guarantee" (I believe) is intended for the Go language itself and not the go tool.

[thepudds]

I think the transition path would need some more thought, and that might influence the details of a proposal here.

@Merovius, one quick comment:

I don't think we can modify the semantics under which an older version of the go toolchain interprets a program. I think go1.16 would fail to compile a Go module written with an "example.com/foo/bar" import and an "example.com/foo" v2.0.0 directive in go.mod, due to a violation of SIV. Which is, maybe surprisingly, a good thing for the design - a compilation error can be detected in the presence of a go 1.17 (or whatever) directive and warn that a newer Go version is needed.

Under this proposal a compile error might help, if it that was the case.

If there is a compile error and the consumed module specifies a go language version in its go.mod that is higher than the toolchain in use, an older cmd/go will report an error like:

note: module requires Go 1.17

However, it depends on how this possible proposed future change is encoded in go.mod -- for example, an older cmd/go might error out before ever trying to compile, or alternatively an older cmd/go might interpret the go.mod from the future cmd/go as an incomplete go.mod and downgrade to v1/v0.

---

Two examples:

Example 1: if a future cmd/go encouraged:

require github.com/peterbourgon/ff v3.0.0       // note no /v3 at end of module path

I think cmd/go 1.16 attempting to consume that module would emit an error before there is any compilation error. Something like:

parsing go.mod: require github.com/peterbourgon/ff: 
    version "v3.0.0" invalid: should be v0 or v1, not v3

Example 2: On the other hand, if a future cmd/go encouraged an import like:

import "github.com/peterbourgon/ff"            // note no /v3 at end of module path

with a go.mod like:

require github.com/peterbourgon/ff/v3 v3.0.0   // note the /v3 at end of module path

...then I suspect cmd/go 1.16 would interpret that as an incomplete go.mod (with a spurious/unused require on github.com/peterbourgon/ff/v3) and fallback to using the 1.16 cmd/go's interpretation of github.com/peterbourgon/ff@latest, which is a v1 release.

---

One possible path forward could be to stagger the roll out. For example, teach go1.17 how to give a graceful error for the planned future go.mod change or have a partial interpretation of the future change, but don't go actually live with the full change until, say, go1.19 to give time to time for older cmd/go to start aging out of the ecosystem. It would come down to the details, though, in terms of how plausible or appealing that might be. Or perhaps teach go1.17 to view a go 1.18 or higher directive in a go.mod as a hard error rather than the softer "give it a try, but give a hint about go version if compilation fails" strategy of today, and make the proposed change here for go1.19 (or whenever). The timing of generics could play into this as well as a separate change that will likely pull people forward on their toolchain version faster than a typical upgrade pace.

Alternatively, the change to go.mod format could purposefully break old cmd/go go.mod parsers, and hope people google whatever error message from older cmd/go to understand that it implies they need to upgrade cmd/go. That might not be a perfect solution, though, in terms of churn and community impact.

Finally, sorry if either of those examples above are off, but if anyone here is inclined, it could be helpful to do some small tests around how current cmd/go might react to a future cmd/go change under this proposal.

[pkieltyka]

^ could this work? .. is it possible? @bcmills @ianlancetaylor @peterbourgon @rogpeppe

• Each module will have its own consistent view. At build time, module linking will reconcile resolution from go.mod down the tree
• Take note of the green arrows, as might be the most contentious idea, but I don’t see this being a problem in practice? What do others think?
• What other failure states exist in the diagram above that have been missed to be considered?
• Please also let me know if anything in the diagram is confusing/unclear. The intention of the diagram is to render a normal Go module tree as we know it to work today, but noted in blue text how it may be consumed from source and how resolution would work at build.
• The intention of this design is to allow a package to be referenced under a full SIV import, or optionally aliased by the highest major defined in the program's go.mod without the "/vX" SIV path.

I also propose in addition to the support above, that go get github.com/some/pkg@latest always resolve the latest major version. Yes this would be a breaking change to the go get command instead of only querying v0 or v1 under that path, it should consider all. This is absolutely worth it for a sound and intuitive developer experience, and happy to hear scenarios where such a change would be problematic to update to, or reasons to not support it.

[pkieltyka]

@Merovius @thepudds thanks for the ideas, but would you mind opening up a separate issue describing your proposal and solution? I understand we're aligned in our end-goals and efforts (thank you so much), but it is drifting from the approach I had been seeking, and it's easier to discuss and evaluate one proposal at a time in this linear and long thread format.

[thepudds]

Hi @pkieltyka, just to clarify, I am not proposing anything. I don't understand the transition path of the current proposal, though.

Under your proposal, what is the experience of someone using Go 1.16?

[pkieltyka]

@thepudds did you have a chance to review the diagram above in https://github.com/golang/go/issues/44550#issuecomment-787455352?

Under the proposal I'm suggesting, the experience of Go 1.16 would be the same as it is today (as I have noted countless times in my posts throughout this thread). My proposal would be non-breaking and identical to as it works today for Go 1.14, 1.15 and Go 1.16.

Hence, why I'm asking @Merovius to open a separate issue as I think his approach is viable and cool too, but it doesn't provide the same compatibility-level in the experience. My proposal is simply additive to a future version of Go, ie. 1.16.x or 1.17. The only breaking change I'm suggesting is with the go get command related to functionality of '...@latest' as noted at the bottom of https://github.com/golang/go/issues/44550#issuecomment-787455352

[thepudds]

Hi @pkieltyka

@thepudds did you have a chance to review the diagram above in #44550 (comment)?

Yes, though always some chance I have not understood it. That said, I do not see that diagram calling out different versions of the toolchain in use.

Under the proposal I'm suggesting, the experience of Go 1.16 would be the same as it is today (as I have noted countless times in my posts throughout this thread). My proposal would be non-breaking and identical to as it works today for Go 1.14, 1.15 and Go 1.16.

If your proposal is implemented in Go 1.N for some future N, what would the experience be for someone using the Go 1.16 toolchain who depends directly or indirectly on a module that was published by Go 1.N, including under the different permutations of how Go 1.N would use or not use SIV under your proposal?

There are different potentially valid answers to that question, but I am not sure what is being proposed regarding that question.

If that is covered above (it's a long thread! 😅 ), I'm happy to re-read, but so far I have missed that.

[Merovius]

@pkieltyka I won't open a separate issue myself. As I said, my intention was to put your proposal into words that are more concrete and IMO likely to converge on consensus. You apparently think I failed in that (i.e. that I didn't reflect your proposal), in which case my comment should just be discarded.

Opening a proposal would imply (to me) that I think SIV should be optional - but I don't and neither do I think it shouldn't be. I was simply trying to steer the conversation towards consensus one way or another.

@thepudds You are correct, I believe, that any design for this proposal should encode the necessary knobs into go.mod in a backwards compatible way. I think there are many ways to achieve that, so I'm not too worried to postpone the conversation about syntax until we've agreed that we want SIV to be optional and what semantics we want to achieve. I don't think such a consensus is even close right now.

[pkieltyka]

@thepudds hmm, yes, its difficult to answer that question -- as either I don't understand how Go modules work, or I haven't communicated my ideas in the diagram above correctly for you to understand. I'll make one more diagram showing before & after, where before is how I view Go modules working today, and after this proposal.

[thepudds]

@pkieltyka one thing you could consider is creating ~2 trivial "library" style public repos that pretend to have been written under some future Go 1.19 (or whatever version) under your proposal, where each one imports the v3.0.0 version of github.com/peterbourgon/ff, and with one repo using SIV and one repo not using SIV to import github.com/peterbourgon/ff. (You could pick whatever target you want, but peterbourgon/ff seems reasonable choice that has seemingly well formed v1, v2, and v3 modules, and its author has been chiming in here).

That said, this is your proposal, so maybe you don't see that as worthwhile, but that would likely add some clarity, including it would enable seeing what Go 1.16 would do today when importing your "future" public repos into a trivial consumer using the Go 1.16 toolchain.

@Merovius FWIW, I'm in the opposite situation 😅 -- I find it easier to imagine the semantics under optional SIV, but I'm so far having harder time imagining how it can be done in a completely backwards compatible way in just one step, though it might just be a failure of imagination so far on my part. Even so, that doesn't mean there is no path forward (e.g., I threw out some options in the last three paragraphs in https://github.com/golang/go/issues/44550#issuecomment-787451360), or a change can be so valuable that it overcomes migration costs.

In any event, a complete proposal eventually should at least comment on migration & compatibility considerations. That could happen sooner or later.

Or, maybe I've just completely misunderstood what @pkieltyka is proposing 😊

[Merovius]

@thepudds To be clear, I was suggesting the syntax of go.mod should be backwards compatible - that is, go1.16 should be able to parse a go.mod written for a go1.N, even if it contains whatever knobs we include to make this proposal possible. I think that's not particularly hard - we just extended go.mod syntax for retract directives, for example. [edit: And yes, maybe it will really take two steps in the end. Again, I just feel confident that we can make it work, so I think there are more pressing issues we need to agree on first]

I agree that it's hard to imagine any way to make SIV optional that would enable you to compile a module taking advantage of that feature correctly with a go1.16 compiler. But as long as we make sure that go.mod is parseable (including its go1.N directive) and that compilation fails with go1.16 if you take advantage of SIV being optional, it can give an error message that go1.N is required.

The other compatibility question is if a module written for go1.16 and a module written for go1.N (opting out of SIV) can be compiled into the same program by go1.N. I agree that that's important to answer as well. I don't know what @pkieltyka's plans for that are, because I still don't understand what he is suggesting for how his proposal should be implemented, but I feel confident that there is at least one way to implement it while maintaining that compatibility, so I'm not too concerned.

[peterbourgon]

I'll attempt to restate the rules to a proposal based on what was suggested by @Merovius, because I think it ticks all of the boxes we're discussing now.

In go1.17 modes and above...

1. The unversioned module identifier is by default interpreted as having no version, rather than major version 0 or 1.
2. Module version suffixes /v0 and /v1 are now permitted and identify major versions 0 and 1 respectively.
3. Module identifiers in go.mod, and import statements in Go source, are now permitted to omit the major version suffix.
4. The specific resolved version of a module continues to be determined by it's entry in go.mod. An unversioned identifier may be associated with any major version, not just major version 0 or 1.
5. ~If a module A depends on multiple major versions of another module B for any reason, at least one of those dependencies must be universally identified in A with a major version suffix, both in go.mod and in import statements in Go source.~
   If multiple major versions of the same module would appear in a single go.mod file, {at most one, none} may use the unversioned module identifier.
6. ~If a module A depends on multiple major versions of another module B, and has assigned the unversioned identifier of B to v1.x.x, and the versioned identifier B/v2 to v2.x.x, other dependencies in A's dependency graph that use the unversioned identifier of B to refer to v2.x.x may continue to do so, under the assumption that this localized knowledge will be parsed from each module's go.mod and preserved somehow.~
   The association of an unversioned module identifier to a specific version is module-local and not transitive.

Am I missing anything?

[icholy]

What would the canonical import path be? Ie the one used in error messages and the symbol table?

[nemith]

I read @Merovius suggestion a bit different and would like a proposal to allow go.mod to control what the behavior of a unversion import. Leave the rest of SIV alone.

The unversioned module identifier is by default interpreted as having no version, rather than major version 0 or 1.

I would want this to keep the normal behavior today unless explicitly set in go.mod. If anything just for the sake of backward compatibility. If not this is a huge breaking change.

Module identifiers in go.mod, and import statements in Go source, are now permitted to omit the major version suffix.

I would rather see it be explicit in go.mod what version you want the "unversioned" import to mean. If nothing defined in go.mod then it means the existing behavior. If you overwrite it you can make the unversioned import port to any version. Maybe include tooling to help change this when go upgrade is issued.

If a module A depends on multiple major versions of another module B for any reason, at least one of those dependencies must be universally identified in A with a major version suffix, both in go.mod and in import statements in Go source.

If a module A depends on multiple major versions of another module B, and has assigned the unversioned identifier of B to v1.x.x, and the versioned identifier B/v2 to v2.x.x, other dependencies in A's dependency graph that use the unversioned identifier of B to refer to v2.x.x may continue to do so, under the assumption that this localized knowledge will be parsed from each module's go.mod and preserved somehow.

Instead just make the non-version override be local module specific only. This still means people need to update thier modules but it fits inside what SIV and modules are trying to produce instead of going around. I think that anything that is trying to affect global state is going to have a bunch of breaking corner cases which defeats the entire purpose of major version and SIV. This in turn would cause more issues and support for immediate version of a module.

This just makes the optics a bit better and solves the "update all the imports" while still keeping the SIV semantics and intentions.

TL;DR Allow a user to override what version an unversioned imports will point to with a directive in go.mod with some tooling around upgrading. This is module local only and doesn't affect other modules.

[thepudds]

Briefly, on the @pkieltyka proposal, it seems to be saying that the proposal addresses compatibility issues. As far as I understand the proposal, it doesn't, including for example there are scenarios where a Go 1.15/1.16 consumer would see unexpectedly downgraded major versions (e.g., a v1 instead of a v3).

If you were to rank solutions from "fully transparent migration & compatibility" vs. "generate a clear error message when needed" vs. "generate a cryptic error message" vs. "downgrade major versions below a required major version", the last one is likely the least desirable of that set. That said, maybe I have a flawed understanding.

For @peterbourgon summary of @Merovius proposal, I would suggest adding some verbiage that exact signal in go.mod is still TBD, and compatibility & migration strategy is still TBD (and in my view, having those as TBD is of course not immediately fatal to a WIP proposal, including I threw out some possible approaches in https://github.com/golang/go/issues/44550#issuecomment-787451360).

[pkieltyka]

@thepudds @Merovius yes you're right, by my proposal it wouldn't be backwards compatible with Go <= 1.16 < 1.x. The Go language and go.mod file are backwards compatible, but as soon as someone adopts the semantics of my proposed alias in a newer Go 1.x version, then older Go <= 1.16 < 1.x wouldn't behave properly as it would expect a SIV path while the source is using an alias of the major.

I also like @Merovius 's proposal and notes Peter Bourgon provided :+1: Similarly it would break for older versions, but I don't see it working any other way. I prefer this concept of "unversioned" import path, as I think it's more clear what is happening in the go.mod and is explicit. Compared to my proposal, I was suggesting to keep the "versioned" import path and just allow for aliasing implicitly, hoping it would achieve a more backwards compatible path, however, it's not so. Given that both require a breaking change, I think @Merovius 's approach has a more clear distinction of "versioned" / "unversioned" import use in the go.mod and arguably the clarity makes for a superior developer experience.

[beoran]

Well, if tooling alone is not sufficient, I do feel the proposal of @Merovius as explained by @peterbourgon is a far clearer approach than what @pkieltyka is proposing, although I don't know whether it can be implemented easily. I can see the appeal of being able to use an unversioned import in Go code and then resolve the version of the dependency in the go.mod file only. This is much like how it is done in other languages, like, say Ruby with unversioned requires and versioned gem files.

[Merovius]

I'm sorry, but

I do feel the proposal of @Merovius as explained by @peterbourgon is a far clearer

I think if you insist on calling it "the proposal of @Merovius" (contrary to my repeated clarification that I was intending to simply clarify @pkieltyka was proposing - even though I apparently did poorly) you should also say "the proposal of @peterbourgon". What he describes differs in several key aspects from what I described, to the degree that I would most likely vote against his design. You can discuss it freely, of course. And you can propose it. And it might get accepted. But I must insist, that you leave my name out of it.

[edit: I guess it serves me right for saying I was trying to clarify the original proposal :) I guess it is accurate to call what I described "my design". But still, what @peterbourgon described is not that, just as what I described is apparently not what @pkieltyka intended]

[peterbourgon]

@nemith

I would rather see it be explicit in go.mod what version you want the "unversioned" import to mean.

This is what I intended to convey. You can omit the explicit version suffix from a module identifier in go.mod, but you still have to provide the explicit version. That is, require github.com/peterbourgon/ff/v3 v3.0.1 is now permitted to be expressed as require github.com/peterbourgon/ff v3.0.1 with equivalent results.

Instead just make the non-version override be local module specific only.

This is also what I intended to convey. Given module A with require example.com/foo v1.0.0, and module B with require example.com/foo v2.0.0, then of course the unversioned identifier is "local" to those modules only. But if some module C imports both A and B, then we're at the situation I describe in point 6. Module C now depends on two different major versions of foo, which is still allowed, but must be disambiguated by giving at least one of them a SIV-style major version suffix. No changes are necessary to module A or B, however.

[peterbourgon]

I agree that it's hard to imagine any way to make SIV optional that would enable you to compile a module taking advantage of that feature correctly with a go1.16 compiler.

New versions of Go routinely introduce features that make them non-backwards-compatible. For example, a project using package embed compiles with Go 1.16 but not with Go 1.15. So I don't see this as a problem.

[peterbourgon]

@icholy

What would the canonical import path be? Ie the one used in error messages and the symbol table?

The "true" or "plumbing" import path should include the major version suffix. Ideally, v0 and v1 should become distinct major versions.

[Merovius]

FWIW, my 2¢ to @peterbourgon's design:

1. The unversioned module identifier is by default interpreted as having no version, rather than major version 0 or 1.
2. Module version suffixes /v0 and /v1 are now permitted and identify major versions 0 and 1 respectively.

I still believe coupling this is counterproductive.

3. Module identifiers in go.mod, and import statements in Go source, are now permitted to omit the major version suffix.

I also still believe this to be a mistake. IMO it is confusing if there are two different ways to write the same import path - a module should be required to stick to one or the other.

5. If a module A depends on multiple major versions of another module B for any reason, at least one of those dependencies must be universally identified in A with a major version suffix, both in go.mod and in import statements in Go source.

You've clarified that by "for any reason", you mean "even transitively", I believe. I don't understand the reason for this requirement. AFAIK, unless A directly depends on B, A does not even need to mention B in its go.mod. This requirement also seems to go against the spirit of this proposal. Because if A directly depends on B@2 and some other transitive dependency C also depends on B@2 but then bumps that to B@3, A now has to go through their source and change all their import paths. This should really just affect direct imports.

In general, I don't like that this design pretty extensively overturns the way versioning in modules works, without any way to opt out of that. That just seems extremely and unnecessarily disruptive. I can get on board with making SIV optional in the sense of giving people an escape hatch to not have to adhere to it - but walking back significant portions of the modules design at this point, just because some people don't like it, is problematic from my POV.

[peterbourgon]

@Merovius

IMO it is confusing if there are two different ways to write the same import path - a module should be required to stick to one or the other.

All else equal, I agree — but all else is not equal here :) Think of it as an alias, which can be used only when it is unambiguous which version it refers to.

. . . This should really just affect direct imports.

You're right that this point may be overly specified. But even if A depends on B only transitively/indirectly, B will still appear in A's go.mod, right? My point with this point was to address the situation where two different major versions of B would be listed in a go.mod — at most one could claim the unversioned module identifier.

I don't like that this design pretty extensively overturns the way versioning in modules works, without any way to opt out of that.

This approach leaves SIV completely in place, and I don't think it walks back any significant part of modules design. Anyone who prefers to continue using the major version suffix, either in the modules they produce or the modules they consume, may continue doing so. The proposal changes the definition of the unversioned module import path to be more permissive, and allows users to opt-in to having it refer to any valid version, including across major versions. And that decision doesn't ripple outwards, it applies only in situ and is overridden by any consumer if they so choose.