Handle many-simultaneous-releases better

fredemmott commented 5 years ago

[x] spawning (number of releases) * (number of distributions) seemed to be fail at the spawning-EC2-nodes step. Need to investigate
[x] update-repos downloads the entire hhvm-scratch S3 bucket, not just the release it needs. This takes longer than the timeouts; it should either just download what it needs for the release in the specified version, or we should (even better) increase the timeouts and handle publishing multiple releases in the same job
[x] macos: we currently have two workers, so we have to do (number of releases) jobs serially
[x] update-repos needs better failure handling
[x] ec2 instances should shut down quickly, after making sure the failures are logged
[x] macos: integrate with EC2 step functions
[x] macos: use the same system for nightlies and multiple-simultaneous-releases

fredemmott commented 5 years ago

We're hitting an instance limit; we can get 44 c4.2x-large, but currently require 56.

Filed a request for an increased limit.

fredemmott commented 5 years ago

request approved, hopefully this is resolved. will test shortly.

fredemmott commented 5 years ago

We're no longer hitting an EC2 node limit, but we are hitting a rate limit for createinstance

fredemmott commented 5 years ago

Workaround: launch in two batches (4 releases at a time). Hopefully #191 makes this unnecessary

fredemmott commented 5 years ago

We now have Azure for handling many macos builds; leaving this open for now though as we really should move the nightlies to that too

fredemmott commented 4 years ago

yeeeep, all done, with @jjergus doing the bulk of this :)

hhvm / packaging