Establish a build pipeline to push raw binaries to S3

[arcrose]

This PR includes a few new things to facilitate a new build pipeline. The goal is to start having TravisCI start produce builds of the MIG Agent and to have those builds uploaded to an S3 bucket for anyone to download. This should help to facilitate a move away from the complicated build and release process we currently follow to produce unique builds for each team at Mozilla.

Changes include:

1. A new mig_agent_cfg tool that produces a mig-agent.cfg file's contents given some command-line flags
2. A new build-agent-release.sh tool that will cross-compile the agent and the mig_agent_cfg tool for each platform we want to be able to deploy to (all six combinations of Linux, Darwin and Windows builds for i386 and x86_64)
3. A new releases.json file within which we will record information about builds of the agent and agent config tool so that consumers can quickly determine the name of the latest release and get a sha256 hash of the binary
4. A new update_release_json tool that will update releases.json given information about a specific release
5. An update to our TravisCI configuration to have it deploy everything built by the above three tools as well as the releases.json file to a public S3 bucket

Note that Travis appears to not push to S3 from branches other than master. So, once this change lands, we'll have to verify that that works.

[ameihm0912]

@zsck just a couple comments here for consideration

There are a few reasons we have avoided cross-compilation in the past:

• Some modules use cgo, so cross-compiling here will not work for them. It looks like there might be a few ways to work around this by installing platform specific build tool chains, but have never gone down this path.
• There are platform specific tests, but this will just run the tests on Linux which may not be the route we'd want to go if the Darwin/Windows builds are going to be deployed.
• Darwin requires a signature on the binary via codesign to properly execute.

[arcrose]

@ameihm0912 If I were to restrict this to just building for Linux, how would things play out?

[ameihm0912]

I think doing the Linux build here would be fine

No reason we can't leave the other builds in here too if that's the route you want to go, just wanted to point those small gotchas out. There will be feature discrepancies in the resulting binaries, so we've avoided this because it isn't a complete solution.

The ideal scenario would be doing builds/tests on the native platforms themselves, but this requires more machinery to do.

[floatingatoll]

The Taskcluster project within Mozilla offers cross-platform build and CI with GitHub auto-integration for specifically the platforms you’re having issues targeting, if/when that becomes a serious obstacle to progress.

On Wed, Aug 22, 2018 at 12:45 Aaron Meihm notifications@github.com wrote:

I think doing the Linux build here would be fine

No reason we can't leave the other builds in here too if that's the route you want to go, just wanted to point those small gotchas out. There will be feature discrepancies in the resulting binaries, so we've avoided this because it isn't a complete solution.

The ideal scenario would be doing builds/tests on the native platforms themselves, but this requires more machinery to do.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/mozilla/mig/pull/463#issuecomment-415156107, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFqDFfzHS680jGMBk0iCP1UYPEJ23vlks5uTbTfgaJpZM4WIEOC .