Run build script on server side.

[sideshowbarker]

Relates to https://github.com/whatwg/html-build/pull/28

This is part of the fix for the problem of making the build service report the correct line numbers when a source file has parser errors. But beyond that, it is part of enabling users to run the build with zero dependencies installed locally other than bash, zip/unzip, and git.

This requires the html/build/build.sh script to be installed on the server and pointed to in the config file (in place of pointing to the wattsi binary).

[domenic]

Can you give a more high-level overview of how this is supposed to work?

Currently as input we take a multipart upload with files source, w3cbugs, and caniuse.

Now it seems like we're taking as input quiet, verbose, and a source file that is a zip file containing... what? And the output is... what? And it's coupled to changes to html-build?

This is a big change all at once :-/. I'd prefer to land this piecewise if at all possible, with a bit more planning before code-writing...

[sideshowbarker]

Can you give a more high-level overview of how this is supposed to work?

So essentially it works by changing the server-side build environment to behave exactly like the local build environment—in particular to expect only a single input, the HTML spec source file.

Currently as input we take a multipart upload with files source, w3cbugs, and caniuse.

This changes things such that we no longer need to upload the w3cbugs and caniuse files but instead we just have the build generate them on the server side in the same way the build generates them in the local case. It makes the server-side build just another local build.

Now it seems like we're taking as input quiet, verbose, and a source file that is a zip file containing... what?

A source file is just an HTML spec source file, as with local builds. Optionally, it can be a zipped source file, but the only reason I added that was just to reduce the size of the upload (from 5.8MB to ~1MB) to save people bandwidth and upload time when building (especially for people who are on slow connections).

And the output is... what?

The output is the exact same wattsi-output tree that a local build generates (and consumes), with a build.log file tucked in that's the log for the build run on the server.

And it's coupled to changes to html-build?

Yes. Though really the only change there was in what we send to the build server. We would now just send the source and let the build server run all the steps need to process that, including the steps needed to get the w3cbugs and caniuse data, and the steps for getting the Unicode CLDR data and the unicode.xml file from the W3C site, etc. And running the perl scripts.

So this also has the consequence/benefit of eliminating the requirement for users to install the Perl XML::Parser module, if they don’t have a local wattsi executable and just want to have the build server handle it all.

Also, altogether in my environment it takes only about 90 seconds to do all this—I mean, to upload the source file to the build server, and to wait for the build run to complete on the build server, and to download the output and do that final steps with that locally (creating the output/multipage dir, etc.)

[domenic]

OK, that makes a bit more sense. If you don't mind, I'd like to take a crack at working on this myself. I'd make the following changes:

• Query strings for options
• Use the body instead of multipart, as you suggested. We can then do zips with content-type switching instead of filename switching.
• Use common cache dir for every build, instead of re-downloading for every user

I am still learning how to use this web framework (koa) so I'd enjoy the chance to do it myself :).

[sideshowbarker]

OK, that makes a bit more sense. If you don't mind, I'd like to take a crack at working on this myself.

Yeah, of course—that would be great. I have little real experience with doing much real-world stuff with node, and don’t even know basic things like what’s the normal way to access query params, let alone stuff like what additional modules might be around to help.

• Query strings for options
• Use the body instead of multipart, as you suggested. We can then do zips with content-type switching instead of filename switching.

Yeah, I think that would be the right way to do it.

• Use common cache dir for every build, instead of re-downloading for every user

Yep—that clearly makes a lot more sense and should also dramatically reduce that server-side build time, and the time that users need to spend waiting for the build to complete.

I suspect it would then be on the same order of reduction in time as a for local builds, where the build time goes from a minute or more down to just 10 or 15 seconds. Of course the difference would be the time required to upload the source file and download the results, and that’s a factor that depends on the speed of the user’s network connection. But I think for the common case of most people have at least 10Mbps+ it should get time down to just 45 seconds—or roughly half of what it would take if we didn’t cache the dependencies on the server side.

[sideshowbarker]

I am still learning how to use this web framework (koa) so I'd enjoy the chance to do it myself :).

And I will enjoy observing how you do it so that I can learn from it :-)