Use Mage instead of Make and clean development tooling

[johanstokking]

Summary:

Our development, build and CI tooling is getting a bit out of hand. Makefiles are complicated and annoying, so we decided to switch to Mage which allows us to write our development, build and CI tooling in Go.

What is already there? What do you see now?

We have simply written too many Make rules that aren't used, or give unexpected results. The dependency graph has more resemblance to spaghetti than to a directed acyclic graph. Variables are overwritten in multiple places in different files.

What is missing? What do you want to see?

I'd like to see a clearer separation in build targets, so that we can split our CI jobs in properly scoped containers:

• Go quality: Testing, Linting, etc.
• JS quality: Testing, Linting, etc.
• Build: Full build with clear dependency flow (first JS, then Go).
• Checking the repo: Regenerate protos, messages, etc. and check for diffs.

The last job ("Checking") is added so that the other jobs can run in parallel. Even though Go tests use .pb.go files, we don't need to regenerate them, because we can assume that any changes would already be built and committed to the repo. The "Checking" job will make sure that this is indeed the case before PRs are merged.

Obviously, the Go tests should not invoke yarn or webpack, because our Go tests can run just fine without the webui hashes.

The other way around could be a bit harder, because Mage requires Go. Therefore we may have to find a nice solution for running our JS quality builds in a container without Go.

How do you propose to implement this?

For a smooth transition we should probably start migrating Make rules to Mage one by one. The old Make rules can just call Mage until everyone is used to Mage.

I think it's important to consider the differences between development and CI. In development environments, both Go and NodeJS are installed, but on CI this may not be the case, as Go and JS jobs could run in different containers (with different images). The mage binary could be built separately of the actual jobs (see below), but it should still avoid calling programs that are specific to one environment (generating protos, calling node_modules/.bin/json, etc.). The simplest way to avoid these kind of problems is to avoid dependencies (so Go testing should not depend on the version/ttn.go being generated by a task that also calls node_modules/.bin/json). We can add more higher-level tasks for development that combine multiple lower-level tasks.

Running JS quality jobs in a node container could be a challenge. Perhaps a multi-stage build with Github Actions instead of Travis could help here. The first action could generate the .mage/run binary, which is then used by the subsequent js quality action.

---

Original issue: https://github.com/TheThingsIndustries/lorawan-stack/issues/712 by @htdvisser

[htdvisser]

I just discovered that @kschiffer added build tooling in #431 that depends on htdvisser/protocontainer, which is an old sideproject of mine, and should really not be used here. Needs to be replaced ASAP.

[kschiffer]

Hmm. As in remove it now, or wait until we found a replacement?

[htdvisser]

I mean remove it as soon as possible.

It's not possible right now, because we don't have the options functionality in thethingsindustries/protoc yet. We've been waiting for https://github.com/pseudomuto/protoc-gen-doc/pull/376 to get merged, so that we can use a tagged version of protoc-gen-doc in our official image, but I think it's time to just fork protoc-gen-doc into @TheThingsIndustries, because it doesn't look like the repo is actively maintained by @pseudomuto.

I'll see if I can create the fork this week, so that we can update https://github.com/TheThingsIndustries/docker-protobuf and get rid of my personal protocontainer.