Use golang/build repo for build/packaging infrastructure

[dagood]

https://github.com/golang/build

This subrepository holds the source for various packages and tools that support Go's build system and the development of the Go programming language.

The golang/build repo has one active branch that is used to build/pack/release all active upstream versions of Go. It has a different balance of responsibility than we're used to in the .NET repos: golang/build creates tarballs, MSIs, etc., rather than implementing packaging inside golang/go.

If we can reuse golang/build, we may be able to be more sure we're keeping up to date with changes to:

• .tar.gz/.zip packaging (which files to include); Go builders (CI matrix configuration)
  • microsoft/go currently has a reimplementation of this logic.
  • Internal OneNote page "Reproducing official Go tar.gz"
  • Internal OneNote page "Upstream testing infrastructure"
• MSI creation
  • microsoft/go doesn't have this at all yet: https://github.com/microsoft/go/issues/29

We can also move a significant chunk of our own Microsoft-specific infrastructure to the new repo. This would reduce the amount of cherry-picks needed between our microsoft/go branches when we do bugfixes or add features that we want to apply to all branches.(See comment.)

---

The challenge is that a lot of the code in golang/build is tightly coupled to the build pool implementation. E.g. the tool that produces tar.gz files assumes it has a connection to a build machine using the agent implemented in golang/build, which uses a lot of GCE APIs. (Caveat: I haven't dug too deeply into whether we can try to swap out the agent implementation to make more parts work locally.)

We will likely need to use a fork model and make changes to make the golang/build repo usable in our context. Some areas are more coupled than others (MSIs are not very coupled!) so perhaps we can pick and choose.

Once we have a fork:

• Then we have to keep up with merge conflicts.
• We need to handle tooling updates from the single master to 1.15 + 1.16 + .... This can probably be done similarly to dotnet/arcade: submit a PR to update the pinned version, and fix up issues as they come up. We need to be careful about security and dependencies, because this tooling will necessarily be involved in producing our signed builds.

As an improvement, consider adding PR validation to the tooling repo that runs a build against each branch of Go. (Careful about making a dependency cycle. Maybe the dependency should float, and the check should be optional.)

[dagood]

I'm creating a repo microsoft/go-infra to store Microsoft-specific shared infra, unrelated to golang/build. I named it go-infra to leave room for a go-build repository if we want to forward with this plan in the future.