Parallelize diff generation using dynos

[bgentry]

This is definitely not ready, and IMO the code is terrible & overcomplicated so far. But it does at least work.

Hoping to iterate on this a bit, so don't worry too much about reviewing it yet. I wouldn't be surprised if I can cut the LoC and complexity significantly.

Mostly I've just been struggling to find the right concurrency patterns, and as-is it's a bad mix of a few different things. Essentially it just needs to:

1. Perform one build at a time
2. Upload the build
3. Register the build
4. Add the needed diffs for that build to a work queue (dchan currently)
5. Run a dyno for each diff that needs to be generated, preferably with a reasonable limit on parallelism (don't want 1000 dyno runs at once)
6. Know when all diffs have been generated successfully so it can then mark that release as current

If you have suggestions on the concurrency patterns, I'm all ears! The code for after step 3 doesn't feel right as it's currently built.

I've tried to make the process idempotent. If a build fails at a particular step, that particular build shouldn't complete, but the others still may. Running the build process again should just figure out what's left to be done and do that.

[kr]

The approach looks basically sane to me. A couple of observations:

• Having diffs in place is not a hard dependency of setting the current version. So if some of the diffs fail, we can still go ahead and make the change. It just means that people's updates will be transferring more data. This shouldn't make much if any detectable difference in the UX, since the update happens entirely in the background. In general, I view the diffs as an optimization, so I'm reluctant to introduce logic that treats them as necessary.
• If a diff fails, it's sufficient to log an error message and continue. This means the code that synchronizes on diffs doesn't also have to know whether they succeeded, it only has to wait until the attempts are done. So you can pass &bgroup to GenDiffs and get rid of dgroup and gendone and the select loop in that function. (Might also have to move the setCurVersion call outside of the main loop, instead do them all together after the diffs have been generated.)
• Just moving some of those big multi-way if statements out of the main loop into functions would help the big-picture structure show more clearly.
• Maybe just rearranging some of those same multi-way statements will make things more readable.

[kr]

Another way to approach this is to write the serial version first. How would you write the code with no concurrency. Once you're basically happy with it that way, then add concurrency.

[kr]

Another thing you can do if you need to collect one or more errors is to pass in a buffered error chan, so you can do up to N unconditional sends. After calling Wait, you can use len on the channel to see how many errors happened.

This may or may not be better. Sometimes I have intuition about which things to use, sometimes I just try a few ways and see which one does the best job of "say what you mean", or looks nicest, or expresses the intent most clearly.

[bgentry]

@kr Let me know if you have any final feedback on this one. I'd like to get it merged in tomorrow.

[kr]

Sure, I'll look at it today. Sorry, I didn't get a notification when you pushed up the latest version of this PR.

[kr]

Looks good to me.

[bgentry]

:v: