Nightly builds

[alexpatel]

I have created a barrelfish user on nomnomnom and added this to it's crontab (following the instructions in the README)

Right now it just runs hake for x86_64 and ARMv7 and compiles for both targets; it will only include the hake/make output in the e-mail if the command fails - it's straightforward to have it do more, I just need to deal with the arch-specific testlists before we can get some automated tests

[ghost]

I think we'll find we want at least some of the features from the ancestor script back, like mailing out diffs instead of full logs.

also I think it would be a good idea to keep the logs instead of dumping them in /tmp.

more importantly, unless I misread the script the output from a failed build goes to the script's stdout and will get mailed to the build user by cron rather than being put in the log to be mailed out... might want to test and/or fix that.

[alexpatel]

I think we'll find we want at least some of the features from the ancestor script back, like mailing out diffs instead of full logs.

Cool, I'll bring in timing + build diff - anything else from that script that is desirable (or even other features in general)?

also I think it would be a good idea to keep the logs instead of dumping them in /tmp.

more importantly, unless I misread the script the output from a failed build goes to the script's stdout and will get mailed to the build user by cron rather than being put in the log to be mailed out... might want to test and/or fix that.

So, I realize it's a little convoluted - the block from lines 82 - 90 will redirect STDOUT/STDERR for the entire script into ~/nightly/builds/{date}/nightly-log.out (MSG_FILE), and then at the end MSG_FILE is put into the body of the e-mail. The /tmp stuff was to allow for selectively appendping the output of the make/hake commands to MSG_FILE only in the case they fail.

[ghost]

Yeah, apparently I misread.

One other thing: you removed the bits that turn off all the Unix locale misfeatures; that's certain to bite eventually, especially on Linux.

Anyway, FWIW here are the main features of the script from goat, some of which are expensive to reimplement:

• the deployed script on test machines autoupdates itself when the upstream checked-in version changes;
• there are specifically two build logs, a full one and the summary, and it's the summary that lands in people's mailboxes;
• it clones a fresh tree every time (this is intentional, to make sure it's a clean copy from upstream);
• it always keeps the last successful build, no matter how long the build's been broken;
• when a build fails, it mails out only the diff from the last successful full build log, not the whole thing, and only the first 1000 lines;
• it archives the build logs in their own repo;
• it reports timings for the test suite runs (and for other things but that's less important);

Some of these we don't need and aren't especially important (the archive of build logs, for example, is only very occasionally useful) but some we do, in particular the thing about mailing out diffs and capping the size. Full build logs are huge and wading through them looking for something different is no fun...

The autoupdating is expensive but it wasn't frivolous when I added it: in this day and age when machines sit around on desks, aren't part of a shared operating environment, and are all different, going around and updating the test account infrastructure on every machine that's running tests is a big hassle, and experience dictates that you can't effectively tell people to do it for you on their machines; they won't get around to it or they'll do it wrong. I suppose the modern-day response to this problem is to keep everything in some central docker config, but that isn't actually a solution, just a level of indirection, and you still have to go around and update every machine when you need to change the setup for running docker.

[alexpatel]

Yep, I see - I just re-wrote from scratch to get a baseline, I will work on merging it into the original script to get these juicy features (even the locale flags...) - the only one I wrote today that made it to this list is the "keep 3 builds around" one.

Auto-updating is definitely very cool, I like how it is done - if the script is in the same tree as the one that the script itself is making builds from, and the script does a git pull dev/master before building, then won't we see any changes to the script the day after they are merged? Not a great response time I guess, but one avoids the wrapper script...

[ghost]

The reason for all the restart logic is to not have that delay... :-)