Closed ianlancetaylor closed 5 years ago
Change https://golang.org/cl/142417 mentions this issue: design: add Go 2 transitions background document
I like this idea.
I read the document and I would like to comment on it as follows:
I feel changes to the Go language that are not backwards compatible should increase the first version number of Go. I am normally not such a fan of semver but in this case it seems appropriate. In practise this will mean that after implementing all planned language changes, we will end up having a Go v5.x or maybe even v27.x, but that is fine, since it makes it very clear that the versions are definitely not backwards compatible.
I don't think redefinitions in the language are as much a problem as you make them out to be. In the case of the size of int changing, this was a backwards compatible change since the size of int is implementation defined, and the users of Go should not write any code that requires it to be of any given size. In this case, and all similar cases where backwards compatibility is retained in a redefinition, increase in the minor version of the is sufficient.
In the case of adding the key word check
, this is in essence a backwards-incompatible change. This, as well as any other non backwards compatible changes should come with an increase in the major version number.
So if my proposal here is accepted, then we will certainly have a go version 2.0, soon to be followed by 3.0, maybe even 4.0 and 5.0. I think that the requirement to bump the major version number will give some pause as to whether or not we really want to make an incompatible change.
I further propose that we modify cmd/go to add a new directive to the go.mod file:
lang goN.M
. This directive sets the language version to use when building this module.
I worry that this would be confusing, as go.mod
already contains a go N.M
statement. That isn't used for much yet, though, so perhaps this could piggyback off that statement. Edit: forgot to link to it: https://go-review.googlesource.com/c/125940/
I wonder if this would add lots of complexity over time to cmd/compile
, as it would have to keep language support for an arbitrarily long number of Go versions. If a package uses a language feature introduced in 1.15, is a lang go1.15
statement in go.mod much better than the already possible go 1.15
statement?
GCC supports 5 different C++ standards from 98 up to 2a, and 3 different C standards from C89 until C11. Code that actually works is actually valuable, and there are many million line legacy projects that cannot be updated easily any more. Some complexity to stay backward compatible is unavoidable but necessary.
@mvdan Unfortunately I haven't been closely tracking go.mod files. What does the go N.M
statement mean and when is it updated? It doesn't seem to be documented.
Looking at the CL it doesn't seem to mean exactly what I'm looking for. It seems to be computed as the upper bound of the go N.M
versions listed in required packages. Using those semantics would prevent a 1.20 package from depending on a 1.25 package while still using the 1.20 language version. I'm certainly fine with using the existing go N.M
line if it will do what we need.
As the background doc says, it's definitely confusing that we use the same names for Go release versions and for Go language versions. I don't know how to avoid that. Fortunately I don't think the names will leak out to regular users very often.
I agree with @beoran that the additional complexity in cmd/compile is manageable.
I really like this idea. A couple of random thoughts:
lang go2.2
, it would always use the latest supported v2.2.x
.go.mod
file to specify the version, or otherwise existing code that hasn't been upgraded to a module would quickly stop working or, depending on what changes were made to the language, silently start working differently.go.mod
is a convenient dumping ground for this kind of metadata, but it's not obvious to me that module scope is the right granularity for the lang
directive. It seems like it would be more useful at package or even file scope, to allow large projects to transition gradually/piecemeal.
You mention Perl 6 and Python 3 as examples of what not to do. It might also be worth discussing Python's __future__
imports and Perl 5's use VERSION
as more successful examples of how to evolve a language.
I don't think __future__
solves the same problem. The point of __future__
is to allow people to start using features that haven't been released yet and allow testing of those features. The point of this proposal is to allow a Go 2 project to import a Go 3 project which imports a Go 1 project and have it just work. That way legacy code doesn't need updating when backwards incompatible changes get made. This should hopefully allow Go to avoid the migration problem Python has had.
It's certainly a different approach, but i do think it tackles the same problem: namely, "how can we add language features without breaking old code?".
Instead of asking old code to tag itself with an old language version, as in the current proposal, Python asks new code to explicitly request the features it needs. There's no reason a priori why this approach couldn't be adopted by Go (although i agree that it probably wouldn't be a good fit).
For one thing, in Python, the transition period during which old code and new code can be mixed is usually only a single release, after which the feature becomes the default; whereas Go would presumably want to extend this for a much longer period of time, possibly indefinitely. In such a case, one can imagine that feature directives would tend pile up in new code, leading to an unergonomic language, which is why i say it probably wouldn't be a good fit for Go.
In any case, it seems worth discussing in the proposal, if only to reject.
@DeedleFake I am assuming that we are never going to change the language in a minor version, so lang go2.2
is going to mean exactly the same thing for 2.2.0, 2.2.1, 2.2.2, etc. If that assumption is wrong, then we may need to use three digits in a language version.
I'm not sure it's precisely accurate to say that code without a go.mod file containing a lang
directive will quickly stop working. It would only stop working if we removed some language feature that that code relies on. But that is going to be a rare step. It's an important case to consider, but we aren't going to remove features that are widely used. That said, you are correct that using the go.mod file assumes that those will become widely used. If that is not the case, we should record the maximum language in some other way.
Which is, of course, what @magical suggests. I'd prefer to use the go.mod file if possible, because every separate piece of metadata is a significant pain point. Even if go.mod is not ideal, I think it's worth using if we can, just to avoid that pain.
@magical I updated the background document CL to include a section on Python and Perl feature selections. I don't think feature selection is a good fit for Go, but I agree that it's worth mentioning.
Maybe a change to the package
statement? Could be something like package main go2
, for example, and a lack of a second argument would be the same as saying go1
. That would increase the granularity and provide a place to quickly check what version you're dealing with. Editors and plugins could then detect the version of existing files when creating new files, and copy it, the way vim-go does already for the package name. One downside, though, would be that updating could be more awkward, as you'd have to change every file in the package.
Personally, I think go.mod
would probably be a better fit, but I figured I'd throw this in there.
more useful at package or even file scope
Building on @magical's insight, here's a //go:
directive listing only backward-incompatible language features used in the file.
//go:lang handle, xyz
That's more descriptive than min and/or max versions. (Does a max version imply dependence on the bugs or performance of that version?) This directive (or its absence) also informs new versions of the compiler to disable backwards-incompatible features when compiling older code.
One line should be enough, since very few backwards-incompatible features are expected. A //go:lib
directive could do the same for the stdlib.
Any annotation in the source code itself is troubling because we ideally want the go tool to be able to record the version being used, one way or another, but we certainly do not want the go tool to modify people's source code.
go.mod could store the tool version that was selected via a feature directive or the module author. That is build metadata, whereas a list of backwards-incompatible features required by a specific file is code metadata.
EDIT:
I think the main drawback to a //go:lang
directive is new boilerplate for Go2 code.
go.mod already recognizes a 'go 1.8' etc line. I think we should use that instead of 'lang go1.8', and have the language apply to the whole module.
I'm on board with cmd/compile accepting -lang and whatever changes are needed to make compilation with one language version be able to import compilations from another language version. (I would hope that exported information would just be written in some 'go superset' instead of having to tag individual packages during export with language versions.)
@rsc using the existing go
line in go.mod files may be OK. (I didn't know about it because it isn't documented.)
The current go line does seem to have a somewhat different meaning than I was envisioning. Right now it seems that if a go.mod file says go 1.21, and I am building with Go version 1.20, I will get an error saying "module requires Go 1.21". What I am proposing here is that that case should just compile without incident. I think it's important that the language version be a maximum required language version, not a minimum. Otherwise, everyone is forced up to the version of Go used by whoever built the newest library that they depend on. That's a fast pull to tip, which is not going to work for organizations that need to be conservative. Pulling to tip is not required in the general case, and we shouldn't require it here.
Right, we need to figure that out - we've started that conversation before of course. :-)
Talked to @ianlancetaylor, @griesemer, and others at proposal review.
The current go
line in the go.mod file means "use this exact version of the language semantics".
For a dependency module with a go.mod file with no go
line, or no go.mod at all, we will assume the last version of go before we start making breaking changes. (Let's call that Go 1.L.) That way, imported old code keeps using the older version and does not break.
For the current module, once we start making breaking changes, if you run a go command in that module tree and there is no go
line, the go command will add a go 1.X
line where X is the current version of go. That way go.mod is "initialized" correctly and people don't have to go out of their way to get new language features.
Having decided for a given module which go version to use, the go command would pass -lang=xxx
to the compiler when compiling a package. (For alternate build systems that don't yet know about -lang, we'd default to the compiler assuming the latest version of the language. But the go command would simply always pass -lang.) Note that this means all packages in one module get a consistent version of go, which might help for being sure it's OK to move code around inside one module.
For a dependency asking for Go 1.(N+1) or later, when the current go version is Go 1.N, one option is to refuse to build. Another option is to try compiling as Go 1.N and see if the build succeeds. If we never redefine the meaning of existing syntax, then it should be safe to conclude from a successful build that everything is OK. And if the build fails, then it can give the error and say 'by the way, this was supposed to use a newer version of Go, so maybe that's the problem.' @ianlancetaylor has been advocating for this behavior, which would make as much code build as possible, even if people accidentally set the go
version line too new. It sounds like this should work - in part because we've all agreed not to redefine the meaning of any existing syntax - so we should probably do it and only roll it back if there is a major problem.
Leaving open for additional comments.
Change https://golang.org/cl/144340 mentions this issue: cmd/compile: add -lang flag to specify language version
+1: reclassifying core std lib vs penumbra +1: -lang compiler flag -1: per file directives
I think more thought needs to be put into how to deal with mixed -lang= versions. Neither solution "refuse to build" or "issue a warning" seems right because: "refuse to build" means collecting packages/modules for a product could become quite involved and difficult for trivial reasons. "issue a warning" is against the go practice of failing to build unless we're sure it works. One option may be to explore introducing binary compatibility at the same time -lang= is introduced. So if -lang= is first available in go1.12, then from go1.12 onwards, a commitment to binary compatibility of packages/modules could be made. In this case, -lang=x in one module and lang=y in another with x!=y could run and a warning could be issued, but only if the caller requests uniformity of lang= or some other constraints on it.
question: lang meaning changes: #24543 changes the meaning of a loop without function calls to include the possibility of pre-emption by the Go scheduler/runtime. How does this proposal relate?
I think it's important that the language version be a maximum required language version, not a minimum. Otherwise, everyone is forced up to the version of Go used by whoever built the newest library that they depend on.
This point also relates to language meaning changes. As long as the language meaning/semantics is not precise, it doesn't make much sense to declare the meaning of go 1.N compatible with go 1.N-1 (which is equivalent to specifying a minimum version). Case in point is #24543, which (while solving a myriad problems, also) changes the semantics to something less expressive and has restrictive implications for lower level systems programming. If the language semantics (memory model, scheduling, relation to system calls) were precise, then backwards compatibility (minimum version specification) would be reasonable. But it's just not the case.
@wsc1 maybe propose runtime.Get/SetPreempt() in a new issue, with milestone 1.12?
@wsc1 There is no intention of supporting binary compatibility. That's not what the -lang
option means. If you are building your program with Go 1.N, then you have to build everything with Go 1.N. We've only ever provided source-level compatibility, and that is all we ever expect to provide.
The -lang
option doesn't mean "build with Go 1.X", it means "build with the language semantics of Go 1.X". Just like the -std
option of GCC or clang.
What @rsc outlined above is: if the module requests a newer language version, try to build it with the current language version. If that works, fine. If it fails, report the error, and also say that it may be due to the fact that the module requires a newer language version. I think that is consistent with what you are suggesting.
@wsc1 You are correct that source compatibility is not going to handle all possible cases. So it goes.
@wsc1 maybe propose runtime.Get/SetPreempt() in a new issue, with milestone 1.12?
Already proposed (as something that would work providing it isn't implemented by means of a pre-empting sys call) in that issue.
Also #21314 I proposed a user compiler directive to replace go:nosplit.
No responses (except a thumbs down :), no idea where the pre-emption stuff is or whether it will be implemented, so timeline is awfully tight for figuring out a solution for 1.12, and I'm not a compiler or runtime expert so...
The
-lang
option doesn't mean "build with Go 1.X", it means "build with the language semantics of Go 1.X". Just like the-std
option of GCC or clang.
One potential difference: I think I can link -std=C11 with std=C99, at least in many cases, provided the exported .h includes both or an intersection.
@wsc1 There is no intention of supporting binary compatibility. That's not what the
-lang
option means. If you are building your program with Go 1.N, then you have to build everything with Go 1.N. We've only ever provided source-level compatibility, and that is all we ever expect to provide.
source level compatibility is good enough for me now, although there are arguments for binary compatibility for the language as a whole.
But I don't see how changes like general class of scheduling algorithm provide source level compatibility. If one argues you shouldn't rely on cooperative scheduling, one could equally well argue you shouldn't rely on loop - pre-emptive scheduling. So every program that pre-emptive scheduling "fixes" may be seen as a bug w.r.t. relying on underspecified language semantics, just as one may see relying on cooperative scheduling as a bug.
(And this message may well get a thumbs down, because I'm suggesting that being in the majority or in a position of influence over the language semantics shouldn't suffice to dictate what is compatibility and what is a bug relying on underspecified semantics. If so, I'll respond in advance: thanks for making my point. A thumbs down on issue 28221 of github simply shouldn't matter: if a user considers it incompatible that's enough for it to be so when the language is underspecified)
Other things which, to me, fall in this class are: memory model (atomic relation to C11 stdatomic or C++ equiv), System calls the runtime uses (when/how).
Changes in these things may well be incompatible with legitimate programs, people, and organisations which assume the compatibility guarantee covers redefinitions of such semantics.
From some comments from a recent video from Guido Van Rossum, he said the mistake he made with python 3 was essentially an underestimation of the size and importance of the python users which were not part of the discussions and planning of the new language stuff. In his case, they were the silent majority that caused so many problems in the otherwise consensus-driven process to bring about python 3. I think it is likely Go is in a similar position; my own problem with the scheduling came about precisely because I was too busy working on end goals with Go to take the time to help drive the Golang process in a more fine-grained way than providing things that work well in Go. Users shouldn't be obliged to be so involved. It should "just work" even if you're doing crazy things with atomics and C or system calls that require precise, uninterrupted sequences, or whatever.
The -lang tag following specific versions is a huge step in the direction of recognizing that these things matter. I think it would help to consider the problem when things don't work across Go versions as something to which the underspecified go semantics contribute.
@wsc1
One potential difference: I think I can link -std=C11 with std=C99, at least in many cases, provided the exported .h includes both or an intersection.
And you will be able to link -lang=go1.12
code with -lang=go1.15
code. There is no difference.
@wsc1 As you say, some aspects of Go execution are underspecified. It's hard to fix.
I do not want this issue to get into a discussion of loop preemption, the memory model, etc. That is the much broader topic of what it means for one version of Go to be compatible with another version. That is not what this issue is about. This issue is about mechanism, not policy. Thanks.
@wsc1 As you say, some aspects of Go execution are underspecified. It's hard to fix.
I do not want this issue to get into a discussion of loop preemption, the memory model, etc. That is the much broader topic of what it means for one version of Go to be compatible with another version. That is not what this issue is about. This issue is about mechanism, not policy. Thanks.
Ok, sorry.
@wsc1
One potential difference: I think I can link -std=C11 with std=C99, at least in many cases, provided the exported .h includes both or an intersection.
And you will be able to link
-lang=go1.12
code with-lang=go1.15
code. There is no difference.
@ianlancetaylor
I am a bit confused as to the proposed mechanism. In C you can link .o files with std=A with .o files with std=B, B!=A, and even different compiler versions (at least much of the time). That means to me that in some sense there is binary compatibility in the C case that is lacking in the Go case, which is fine with me but still different enough to distinguish from -std= in C compiler tools.
you also said
That's not what the -lang option means. If you are building your program with Go 1.N, then you have to build everything with Go 1.N.
The main questions I have that would clear up the mechanism for me are:
do you mean you can compile and then link with lang=go1.12 and lang=go1.15 rather than only link already compiled output?
do you mean that in order to compile and link with lang=go1.12 and lang=go1.15 one must use only one version of the go toolchain, rather than say go1.12 and go1.15 respectively?
do you mean that code under lang=go1.12 runs according go1.12 semantics while code with lang=go1.15 runs according to go1.15 semantics all inside one program?
sorry if the answers to these questions are self evident to others.
If the answer to the last question is "no", then the mechanism seems misleading. If the answer is "yes", then we have a whole new class of mixed language semantics on our hands. If the answer is "yes and no, it depends" then the mechanism seems quite complex to me. But in any case having -lang=goA.B seems to me like a good mechanism on which to base moving go forward across versions.
do you mean you can compile and then link with lang=go1.12 and lang=go1.15 rather than only link already compiled output?
Yes.
do you mean that in order to compile and link with lang=go1.12 and lang=go1.15 one must use only one version of the go toolchain, rather than say go1.12 and go1.15 respectively?
Yes.
do you mean that code under lang=go1.12 runs according go1.12 semantics while code with lang=go1.15 runs according to go1.15 semantics all inside one program?
Yes.
Discussion on HN: https://news.ycombinator.com/item?id=18297162
I'm still skeptical that go.mod
is the right place for the lang
directive.
In the modules proposal, Russ wrote,
Our article “About the go command” explains:
An explicit goal for Go from the beginning was to be able to build Go code using only the information found in the source itself, not needing to write a makefile or one of the many modern replacements for makefiles. If Go needed a configuration file to explain how to build your program, then Go would have failed.
It is an explicit goal of this proposal's design to preserve this property, to avoid making the general semantics of a Go source file change depending on the contents of go.mod. With semantic import versioning, if go.mod is deleted and recreated from scratch, the effect is only to possibly update to newer versions of imported packages, but still ones that are still expected to work, thanks to import compatibility. In contrast, if we take the Cargo approach, in which the go.mod file must disambiguate between the arbitrarily different semantics of v1 and v2 of my/thing, then go.mod becomes a required configuration file, violating the original goal.
In this paragraph, Russ is adamant that the semantics of a Go program must depend only on the source code, not any additional files. In designing Go modules, he found it important to preserve this property, so that the semantics of a program does not depend on the library versions defined in go.mod
, even though this goes against standard practice in other languages.
It seems to me that if a lang directive were added to go.mod
as described in the proposal, this property would be lost. The purpose of the lang
directive is to change the semanics of Go programs. (For example, changing the meaning of the "check" and "handle" identifiers, as proposed in the Go 2 Error Handling draft.) If information about the Go language version resides in the go.mod
file, it would not be possible (in general) to look at the source text and know what meaning to assign to it. It would not be possible to delete the go.mod
and still have a working program, if that program depended on old language features.
That said, it sounds like Russ is on board with putting it in go.mod
so maybe this is a non-issue.
Ways to indicate language version in the source. Without any of these Go1 is selected.
//go:lang 2 // 1, 2, etc
//go:lang a, b // enabled backward-incompatible features
package main v2 // less boilerplate
package main (a, b)
Since all backward-incompatible features probably won't land in the same release, we'd later see lang 3
.
Still scratching my head that Go1 didn't reserve any keywords for future use...
@magical A foundation of this approach, as described in the background doc linked in the proposal, is that there are no redefining changes. That is, it is never the case that a program that is valid in version 1.N is also valid in 1.N+1 but means something different. So while we are clearly stretching the sense in which we build a Go program using only the information in the source, I don't think we are breaking it. It should never be the case that using two different language versions cause the program to build and behave differently. The largest possible difference should be that the program builds with one version and fails to build with the other version. That means that a language version aware parser could examine the program and determine the set of versions that could be used the build the program, and we could use that instead of the version recorded in the go.mod
file. That would be a tedious approach, and somewhat confusing for the user, but it means that there is no information required outside of the source code itself.
@networkimprov Any approach that requires an annotation to use newer versions of the language is going to be very confusing for all new developers. I'm not aware of any language that works that way.
@ianlancetaylor, but the pacakge main v2
syntax could work, right?
This assumes, as @networkimprov pointed out, at language removals (or now even redefining changes) only appear on major version updates. In theory you could allow package main v1.20
of course, but for similar reasons that only the major version is part of the go module import path, major versions seams to make sense.
While languages like C++ and C have a hugely successful backwards compatibility scheme, they also accumulate language complexity, making them incrementally harder to learn from scratch. One of Go's selling point is simplicity, that's never been a selling point for C++.
Any approach that requires an annotation to use newer versions of the language is going to be very confusing for all new developers. I'm not aware of any language that works that way.
Maybe, but it's pretty common in config files, e.g. docker-compose
, to start with a version N.M
declaration, that determines how the tooling should interpret the file. Those tools are used by the same developers, to some extent, and if it allows removing pitfalls from the language by allowing redefining changes for common pitfalls, the time they are confused about the first line of the program could quickly be saved in.
Also, this isn't important, but it would maybe less confusing if the suffix is go2
rather than v2
, as I then clearly see that it has something to do with language versions, and not versions of the package.
PS! This implies that selection of the language version is done on a per package basis, not per module. This would allow for a hybrid module to contain both go1
and go2
code, even when the go1
code contains language features removed in go2
.
This could in turn help avoiding a Python 3 / Perl 6 scenario, as you can update your (large) project one package at a time, and don't worry about it, and without adding sub-modules for the go1 v.s. go2 packages.
@ianlancetaylor Thank you for the explanation. That eases my concern.
Change https://golang.org/cl/147280 mentions this issue: cmd/go: add goversion environment variable to testing script language
Change https://golang.org/cl/147282 mentions this issue: cmd/go: add go mod edit -go flag
Change https://golang.org/cl/147277 mentions this issue: cmd/go: add cmpenv command to testing script language
Change https://golang.org/cl/147278 mentions this issue: cmd/go: pass go language version to cmd/compile
Change https://golang.org/cl/147281 mentions this issue: cmd/go: add go statement when initializing go.mod
@rsc wrote:
For the current module, once we start making breaking changes, if you run a go command in that module tree and there is no
go
line, the go command will add ago 1.X
line where X is the current version of go.
What will it add if one is using a tip/master version of go (like go version devel +fe2c588b1c
)?
What will it add if one is using a tip/master version of go (like go version devel +fe2c588b1c)?
The next release. More precisely, the highest numbered release tag that is treated as a matching build constraint as described at https://golang.org/pkg/go/build.
This is implemented.
Change https://golang.org/cl/151358 mentions this issue: [release-branch.go1.11] cmd/go: don't fail if requested Go version is later than current one
Please don’t use any form of “language” to describe this.
The language is Go. We’re all clear on that, right?
How about grammar instead?
As part of moving toward Go 2 as initiated at https://blog.golang.org/toward-go2, I propose some changes to cmd/compile and cmd/go. For background see https://github.com/golang/proposal/blob/master/design/28221-go2-transitions.md.
I propose that we change cmd/compile to take a new
-lang
option. This option will take an argument which is the name of a Go language version. Go language versions use the same naming scheme as Go release versions and Go release build constraints: "goN.M". For example,-lang=go1.12
.This new option will direct the compiler to compile the code using the specified version of the language. It is an error to specify a future version of the language. All versions of the language from
go1.12
forward must be supported.Note that this may need to be recorded in the export data when describing any inlineable functions.
Since we don't know what language changes, if any, will be in the 1.12 release, it's hard to give a real example. But, suppose we had this feature for the 1.9 release, when type aliases were introduced to the language. Then passing
-lang=go1.8
to the compiler would give a syntax error for any code that used a type alias. Passing-lang=go1.9
would permit type aliases as is usual today.I further propose that we modify cmd/go to add a new directive to the go.mod file:
lang goN.M
. This directive sets the language version to use when building this module.When updating a go.mod file, if the file has no
lang
directive a directive is inserted with the version of the go tool. Otherwise thelang
directive is unchanged (unless of coursego mod edit
is used to change it).When building a module, if the go.mod file has a
lang
directive, and it specifies a version N.M that is before or equal to the version of the go tool doing the building, then when invoking the compiler for this module the go tool will pass the option-lang=goN.M
. In other words, if the module records that it was built using an older Go release, a newer Go release will build the module using the language version of the older release.The rationale for these changes is explained in the design document mentioned above.