Distribute build tools as a single binary

[jml]

Background

build-tools is a collection of shell, Python, Go, and Terraform scripts that represent build tooling common to Weaveworks projects, proprietary and open source.

It is currently embedded in each of our projects via git subtree or git submodules.

Problems

Distribution

Both git subtree and git submodules are problematic. They are unfamiliar commands to most git users, which introduces a level of friction with both new & existing hires.

When we make an improvement to build-tools, we face the problem of manually identifying all repositories which use it and then filing a PR one by one. This becomes more and more difficult the more we expand.

Language

Writing robust shell code is hard, even with shellcheck. Writing shell code that communicates intent is even harder.

Solution

• Rewrite everything in Go
• Have a single, top-level binary wbuild with subcommands that make all of the required build and lint functionality available
• Continuously release this binary to GitHub
• Update all projects that use build-tools to
  • remove the subtrees
  • download the latest binary as part of their CI

Pinning

While projects should just fetch the latest build-tools, we should ensure that each master commit has a downloadable binary so that projects that need to pin to older versions can.

Things that cannot be written in Go

Provisioning

We have a collection of Terraform files for provisioning test machines.

This should be moved to Weave Net repository. If making it public is impractical, then it should move to its own private repository.

[awh]

[jml]

I love Haskell, it and Turtle would be a good fit for this. It has much better concurrency support, better error handling, and a mature library ecosystem.

On the other hand:

• We don't have huge amounts of Haskell expertise in the office. People would have to learn, and Haskell's learning curve is steep.
• Our build support for Haskell is not as developed as that for Go, and compile times are slower.
• Building a single statically-linked executable takes care. I'm not aware of an equivalent flag to CGO, and I know from personal experience that statically linking C libraries on macOS in particular is remarkably fiddly. This is why difftodo is not a single executable, and what prompted my initial foray into Bazel. Nothing would please me more than to be shown wrong in this.

If we're not going to use Go, I'd consider using Python. It has the advantages of concision and error handling, and we might be able to freeze things into one executable. It's been over a decade since I've tried that, so maybe the technology has improved. I would anticipate similar C linking issues as with Haskell on macOS.

[awh]

[awh]

Jono also mentioned this in Slack, which is an interesting hybrid Go/Shell approach: https://jvns.ca/blog/2017/07/30/a-couple-useful-ideas-from-google/

[marccarre]

What would be the options/arguments for wbuild? I'm particularly interested in the CLI to lint our projects, as I'm about to improve this and, therefore, re-write parts of it in Go, in order to converge towards a resolution of this issue.

[awh]

Possibly this is interesting/relevant: https://github.com/magefile/mage