all: update standard-library dependencies at the start and end of each development cycle

[bcmills]

The x dependencies vendored into the standard library (via src/go.mod and src/cmd/go.mod) should be updated for each code freeze, so that we can apply any needed fixes to those dependencies without also pulling in unnecessary changes or requiring significant backporting work (see, for example, #36851).

This task, like #11811, #12042, and the API audit (#36167, #32813), should occur regularly in each development cycle. Probably we should update the dependencies at the beginning of each code freeze, and again when we reopen the tree at the end of each code freeze.

CC @golang/osp-team

[gopherbot]

Change https://golang.org/cl/217517 mentions this issue: all: update module dependencies

[dmitshur]

This issue is currently labeled as early-in-cycle for Go 1.15. That time is now, so friendly ping.

[bcmills]

Completed the early-in-cycle update for the 1.15 cycle. Removing the label so we remember to revisit at the close of the cycle.

[gopherbot]

Change https://golang.org/cl/231657 mentions this issue: all: update module dependencies

[dmitshur]

Completed the start-of-freeze update for the 1.15 cycle in CL 231657. Moving to 1.16 milestone and adding early-in-cycle.

[gopherbot]

Change https://golang.org/cl/234002 mentions this issue: cmd: update golang.org/x/mod to v0.3.0 (same commit)

[dmitshur]

This issue is currently labeled as early-in-cycle for Go 1.16. That time is now, so this is a friendly ping so the issue is looked at again.

[dmitshur]

I'm creating a short program that performs this task. Using a program makes it easier to precisely define our update policy, and adjust it over time as needed. It also makes this task less error prone and easier to review. Finally, this can help with #41409 and #36852.

The policy I'm implementing right now is informed by the strategy we used in past manual CLs (e.g., CL 231657), with small adjustments. The policy is for all modules in the GOROOT/src directory (currently, std and cmd), update all direct (not indirect) modules with the import path prefix "golang.org/x/" to their latest tip pseudo-version (which can later become tagged as a release version).

The small adjustment is that the github.com/ianlancetaylor/demangle module won't be automatically updated as part of this process. My rationale is that it's different enough that it probably warrants being updated on its own schedule, when there is an explicit need. /cc @ianlancetaylor

Indirect golang.org/x dependencies are not updated, although they could be. I'm going with the more conservative choice for now. I'm open to feedback on this, and can see pros/cons to both strategies.

I've also considered but left out other modules in the GOROOT tree (e.g., GOROOT/misc) for now.

[dmitshur]

Indirect golang.org/x dependencies are not updated, although they could be. I'm going with the more conservative choice for now. I'm open to feedback on this, and can see pros/cons to both strategies.

Looks like we did update indirect dependencies last time. If we don't, it leaves much fewer dependencies to update, because the go.mod files of golang.org/x modules aren't being updated very often.

Will think more about it and likely go with what we did last time as the next step here.

[gopherbot]

Change https://golang.org/cl/255859 mentions this issue: misc/update: add program to update golang.org/x dependencies

[gopherbot]

Change https://golang.org/cl/255860 mentions this issue: all: update vendored dependencies during Go 1.15 development

[gopherbot]

Change https://golang.org/cl/256357 mentions this issue: cmd/updatestd: add program to maintain standard library modules

[gopherbot]

Change https://golang.org/cl/256538 mentions this issue: cmd: update github.com/ianlancetaylor/demangle

[dmitshur]

CL 256357 has updated golang.org/x/... packages, and CL 248499 (/cc @hyangah) will update the remaining packages (pprof and demangle).

We'll need to repeat this at start of code freeze for Go 1.16, so removing early-in-cycle label.

[gopherbot]

Change https://golang.org/cl/248499 mentions this issue: cmd/vendor,cmd/pprof: sync pprof@1a94d8640e99

[gopherbot]

Change https://golang.org/cl/260538 mentions this issue: cmd/vendor: sync pprof@v0.0.0-20201007051231-1066cbb265c7

[gopherbot]

Change https://golang.org/cl/266898 mentions this issue: all: update vendored dependencies for Go 1.16 release

[gopherbot]

Change https://golang.org/cl/267103 mentions this issue: all: update dependency on golang.org/x/sys and regenrate Windows syscalls

[gopherbot]

Change https://golang.org/cl/269118 mentions this issue: all: update vendored dependencies for Go 1.16 release (alt)

[gopherbot]

Change https://golang.org/cl/275446 mentions this issue: all: update dependencies with changes in scope for Go 1.16 freeze

[zikaeroh]

@dmitshur Not sure this is the right place to ask, but do you know why pprof is getting skipped over in all of these dependency updates? I submitted a PR there before the freeze and was exited to see my fix show up in the next Go release, but the most recent update pulled an older version before my submission (and it's been skipped since).

[bcmills]

@zikaeroh, our automated scripts only pull in changes to the golang.org/x repos, since those are the only packages that are subject to the Go release cycle.

More generally, though: if you make a change that you want to have incorporated in a Go release (in an x/ repo or otherwise), you should also send a CL to update the go.mod file in the main repo as soon as the fix lands (or is released) upstream.

[zikaeroh]

I see. Is it too late now to update the vendored copy of pprof? I'm happy to submit a CL to do this (if that's easiest).

[dmitshur]

@zikaeroh pprof was updated mostly recently for the Go 1.16 release in CL 248499 CL 260538. Are the changes which you'd like to pull in since then in scope of the Go 1.16 code freeze? It's too late for new features and risky changes, but if it's bug fixes then it may be in scope (depending on how risky they are).

[zikaeroh]

It was a fix for google/pprof#515, but I don't know if that qualifies as a bugfix or an enhancement. I can live with building pprof directly and not using go tool pprof if it's too risky.

I was just personally disappointed to see the update CL merged with a version a couple of weeks older than the submission date. I didn't comment at the time as I assumed that CL was a belated pull for the beginning of the cycle, and that there would be another pull for the end of the cycle (which didn't happen).

Note that the most recent pull was actually CL 260538 (which is why I assumed what I did via its commit message). CL 248499 came a few weeks before that.

[dmitshur]

@zikaeroh That bug fix looks reasonable. Please feel free to send a CL updating it (similar to CL 260538), and make me and @hyangah its reviewers. We're still before beta 1, so if the changes look safe to land now, we'll do so (we can revert by beta 2 or RC 1 if it causes significant problems). If the changes look too invasive, it will need to wait for Go 1.17, unfortunately. This is a cost of having a 6 month release cycle—there will be changes that just don't quite make the cut and need to wait.

[gopherbot]

Change https://golang.org/cl/275513 mentions this issue: cmd/vendor: sync pprof@v0.0.0-20201203190320-1bf35d6f28c2

[dmitshur]

By now, the minimum required updates have been done at the end of the Go 1.16 development cycle (via CL 266898 and CL 260538). There are some more updates we need to pull in in order to fix known issues (e.g., issue #31192), but that's tracked by other issues.

Moving this recurring task to the next milestone (Go 1.17) and adding early-in-cycle label.

[gopherbot]

This issue is currently labeled as early-in-cycle for Go 1.17. That time is now, so a friendly reminder to look at it again.

[gopherbot]

Change https://golang.org/cl/298854 mentions this issue: all: update vendored dependencies for Go 1.17

[gopherbot]

Change https://golang.org/cl/296789 mentions this issue: all: update golang.org/x/* dependencies

[gopherbot]

Change https://golang.org/cl/315831 mentions this issue: all: update vendored dependencies for Go 1.17

[gopherbot]

Change https://golang.org/cl/316110 mentions this issue: cmd: update x/arch to pull in CL 315572

[gopherbot]

Change https://golang.org/cl/316250 mentions this issue: cmd: update x/term to pull in CL 316112

[gopherbot]

Change https://golang.org/cl/316251 mentions this issue: all: update x/net to pull in CL 316129

[gopherbot]

Change https://golang.org/cl/316450 mentions this issue: all: upgrade x/text to its latest release version (v0.3.6)

[gopherbot]

Change https://golang.org/cl/316489 mentions this issue: cmd: update x/mod to pull in CL 316113

[gopherbot]

Change https://golang.org/cl/316490 mentions this issue: cmd: update x/text to pull in CL 315571

[gopherbot]

Change https://golang.org/cl/316451 mentions this issue: all: update x/crypto to pull in CL 316109

[gopherbot]

Change https://golang.org/cl/316452 mentions this issue: cmd: update x/tools to pull in CL 315570

[toothrot]

We may want to change updatestd to create a CL per module to improve bisection results.

[cagedmantis]

This issue is currently labeled as early-in-cycle for Go 1.18. That time is now, so this is a friendly ping so the issue is looked at again.

[gopherbot]

Change https://golang.org/cl/347190 mentions this issue: all: update vendored golang.org/x/crypto

[gopherbot]

Change https://golang.org/cl/347191 mentions this issue: all: update vendored dependencies for Go 1.18

[gopherbot]

Change https://golang.org/cl/347189 mentions this issue: all: update vendored golang.org/x/sys

[gopherbot]

Change https://golang.org/cl/348409 mentions this issue: all: update vendored golang.org/x/tools

[gopherbot]

Change https://golang.org/cl/348410 mentions this issue: cmd/pprof: update vendored github.com/google/pprof

[dmitshur]

With the 5 CLs above, we're in good shape for early in cycle. Removing that label to revisit this closer to the code freeze.

[gopherbot]

Change https://golang.org/cl/361094 mentions this issue: all: update vendored dependencies for Go 1.18 release

[gopherbot]

Change https://golang.org/cl/361294 mentions this issue: cmd/pprof: update vendored github.com/google/pprof