Create a Homebrew Tap for macOS ponyc and dependencies

[killerswan]

Given the Homebrew project's hosting goals, I've had a few conversations now about what Pony should do for macOS dependencies and to provide ponyc.

After eyeballing the demo project and beginning to explore what goes in a Tap and how binary Bottle packages work, I have some thoughts about the overall system I'd like us to set up for Pony.

I think the thing to do is emulate how JuliaLang does it:

• They have a tap which provides formulas for (suffixed) versions of their primary dependencies here.
• Those formulas also refer to bottles which they host themselves. We could use Bintray, as homebrew-core does. (Julia is building bottles here and here.)
• For building Julia itself they use their own tap for dependencies here.
• They provide Julia itself in their tap, too. Either the latest stable version, or head (which requires an additional flag to homebrew).

So, the things to set up:

1. a tap with formulas for dependencies (starting here and ci),
2. a Travis config to build and deploy bottles to Bintray,
3. an update to the Pony Travis YAML to use our tap,
4. a formula for ponyc, too.

Out of scope right now: .DMG packaging, Cask, or Pony library packaging.

[killerswan]

It may be interesting to note differences between what upstream (homebrew-core) and Julia formulas look like, so here's a diff for LLVM 3.7, and for LLVM 3.9. I'm inclined to keep closer to upstream for now, and not rename the packages, yet.

[SeanTAllen]

I'm inclined to go in whatever direction you think is best. ;)

[killerswan]

Note, Travis supports these macOS and Xcode versions with osx_image (see here):

• xcode8.2 and macOS 10.12 (Sierra)
• xcode8.1 and macOS 10.12 (Sierra)
• xcode8 and macOS 10.11 (El Capitan)
• xcode7.3 and macOS 10.11 (El Capitan) <-- DEFAULT
• xcode6.4 and macOS 10.10 (Yosemite)

[killerswan]

Current status: the syntax of JSON escaped regular expressions is driving me nuts. If I understand correctly the trick is to make a pattern which makes this function happy. So, a file path followed by a regex match group, for example:

"files": [{
    "includePattern": "/Users/travis/build/killerswan/homebrew-pony/(.*.bottle.*.tar.gz)",
    "uploadPattern": "$1"

Anyways, whether I understand it right or not, I can upload files now to here!

Stand by for more!

[killerswan]

Question in my head right now: what's the easy way to manage rebuilds and revisions? Without replicating all of Homebrew's test-bot system?

I feel like what I have now might be halfway manageable, but we'll have to bump revisions manually all the time to get new builds uploaded, hmm.

For example, this commit was built (# 40) in this Travis job and deployed bottles for libressl and pcre2 to this project on Bintray.

[Bintray Upload] Uploading file '/Users/travis/build/killerswan/homebrew-pony/libressl-2.4.5.sierra.bottle.1.tar.gz' to libressl-2.4.5.sierra.bottle.1.tar.gz
...
[Bintray Upload] Uploading file '/Users/travis/build/killerswan/homebrew-pony/libressl-2.4.5.sierra.bottle.1.tar.gz.snippet' to libressl-2.4.5.sierra.bottle.1.tar.gz.snippet
...
[Bintray Upload] Uploading file '/Users/travis/build/killerswan/homebrew-pony/pcre2-10.22.sierra.bottle.1.tar.gz' to pcre2-10.22.sierra.bottle.1.tar.gz
...
[Bintray Upload] Uploading file '/Users/travis/build/killerswan/homebrew-pony/pcre2-10.22.sierra.bottle.1.tar.gz.snippet' to pcre2-10.22.sierra.bottle.1.tar.gz.snippet

Those so-called .snippet files contain this, pulled out of brew bottle's output, which could be used for a subsequent commit to add this to the Formula (and stop builds of more Bottles):

./libressl-2.4.5.sierra.bottle.1.tar.gz
  bottle do
    rebuild 1
    sha256 "51c4c1f8cc7cc283742e46c09b6d345971fbefb2e1b56256eb79abc9a5399ef0" => :sierra
  end

That's nice but leads to thoughts:

1. I don't know whether the ".1." suffix in the file names comes from the revision or the rebuild number, incremented how or why. Maybe don't run brew test-bot here? (update: nope, no idea) ⚠️
2. I'd like to have failed the subsequent build on deployment collision, so we don't (a) overwrite files or (b) erroneously think we've uploaded new bottles when the job succeeds. We can do this in the main script. ✅
3. I need to expand the matrix to more OS versions. ✅
4. A better extension that .snippet would let me open the files easier... ✅
5. The bottle builds (and merges to create them) must be done on a staging branch, so master always causes actual homebrew users to get binary downloads, instead of occasional source rebuilds. ✅

[killerswan]

After a few more tweaks, here is a model pull request: https://github.com/killerswan/homebrew-pony/pull/5

The process looked something like this:

• [x] Make a request containing the version to build, removing old bottle SHAs.
• [x] Review the code.
• [x] Confirm that bottles were built OK (though not deployed) on Travis.¹
• [x] Admin merge to staging to build and deploy bottles to bintray.²
• [x] Admin make another commit inserting those SHAs in the formula.
• [x] Merge to master.

Note 1: the first binary builds are for any user pull request that has a formula without bottle SHAs. Note 2: the final binary builds are for admin merges to staging to deploy. None is needed on merge to master: all the formulas there should have already been bottled.

And the commit history now like this:

homebrew-pony $ git ll --oneline -5
*   a8a9f97 (HEAD -> master, kevin/master) Merge branch 'staging' for pull req. 5 (pcre2 10.22)
|\  
| * 7e0dc7b (staging) Update SHAs for PCRE2
| * 0a28b32 (kevin/staging, pcre2) Continue to list files
| * f842f3c (kevin/pr/5/head, kevin/pcre2) Start building on staging (also: pcre2 10.22)
|/  
* 5ac1a47 Simplify

[killerswan]

The new process, as I'm noting in the homebrew-pony tap's README:

Formula update process

Here are some checklists to follow when adding or updating a formula, so new binary bottles are created.

The submitter should:

- [ ] Make a request containing the version to build, removing old bottle SHAs.
- [ ] If this is a new formula, update `.travis.yml`, too.

An admin should:

- [ ] Confirm that bottles were built OK (though not deployed) by Travis CI.
- [ ] Review the code.
- [ ] Merge to _staging_ so bottles are built AND deployed to Bintray.
- [ ] Make another commit inserting SHAs into the formula.
- [ ] Merge to master.

This means that for a pull req. the first bottle builds (with unreviewed code) won't be deployed to Bintray. But master will still (due to staging) always have up-to-date SHAs. If master always refers to up-to-date bottles, users will have fast binary installs!

[killerswan]

Hmm, scratch most of that.

The problem with rebuilding everything is that LLVM rebuilds (a) take a long time and (b) seem unnecessary. So an easy way forward is to just not do any rebuilds, and simply mirror! That's what I've now set up https://github.com/killerswan/homebrew-pony to do.

Basically:

• killerswan/homebrew-pony is a "tap" containing formulas, AND
• its Travis config runs a script which checks for mirrored bottles, and (if necessary) downloads them from Homebrew and uploads them to Bintray.

As discussed, then usage is as simple as:

brew install killerswan/pony/pcre2

And the formula update process (as described in the README:

The submitter should:

- [ ] Make a request containing an updated copy of a formula from homebrew-core.
- [ ] Modify it's root_url to refer to our bottle mirror: https://dl.bintray.com/killerswan/bottles
- [ ] Confirm mirroring and usage after merge to master.

An admin should:

- [ ] Confirm that bottle SHAs are OK and Bintray descriptor files look good on Travis CI.
- [ ] Review the code.
- [ ] Merge to master.

Periodically, an admin should:

- [ ] Manually remove any binaries Pony no longer needs to keep.

[killerswan]

A problem right now: upload size limits to Bintray. The Homebrew project has permissions to upload larger files than I can mirror: (see logs here):

[Bintray Upload] Uploading file './llvm@3.9-3.9.1.sierra.bottle.tar.gz' to llvm@3.9-3.9.1.sierra.bottle.tar.gz
[Bintray Upload] Bintray response: 413 Request Entity Too Large. 

Without being able to use a root_url / or mirrors of bottles other than Homebrew's upstream one, I'm not ready to make the next jump from mirroring to arbitrarily-renaming-versions, i.e., to create an llvm@4 formula.

That leaves us with the fairly simplistic situation where we can do homebrew builds with the latest LLVM, and when that latest LLVM version changes we can keep our tap pinned to an older one (and downloading the bottle from Homebrew's repository)... That is, we can point our killerswan/pony/llvm formula at the default set of bottles, no problem, and leave it pointing at 4.0 for a while when 5.0 is released (even without our own bottle mirror, since Homebrew is keeping old bottles around here, even though they are talking tough about removing old formulas.)

I feel like I need to draw some graphs to explain this, though. And to explain what I wanted (or still want, and may poke and prod at S3 to get closer to). 😝

(Incidentally, the Homebrew tools use too much magical inference to decide which version and which file and so on of everything to use, so those llvm@3.8 style formulas are not just a reference to the old version but must build a whole new set of binaries and everything, so that's why they're getting antsy, it's a pain in the butt! And manually checking out whatever commit of the homebrew repo is not actually the worst thing, given the architecture. Did you hear about that concrete bridge catching fire in Atlanta?)

But I'm getting impatient and it's time to go goof around with the upstream LLVM project's macOS binary package (here) and see if we can just bypass Homebrew's LLVM altogether...

[killerswan]

Yeah, here's a way to use upstream's LLVM to build ponyc: https://github.com/ponylang/ponyc/pull/1788