Better scripts

[michaelmwu]

Generation and execution scripts, delite and delitec commands

[michaelmwu]

gen script should be able to take lms dependencies that aren't the source code folder so people don't have to compile it

[xeno-by]

Today I've implemented a prototype of the required scripts. The prototype solution consists of 4 files: delitecfg, delitec, delites and delite. upd. These scripts are now obsolete, since I've pushed their release versions to this repo at https://github.com/stanford-ppl/Delite/tree/develop/scripts.

delitecfg: asks the user for the location of Delite and infers everything else from that. After a successful run it produces a ~/.delitecfg file (example). Delitecfg uses "sbt project-console" to acquire and inspect runtime instances of SBT projects. It is capable of finding out such things as the location of suitable scala interpreter, compiler and libraries, as well as the main compilation paths of the projects that constitute Delite.

upd. Currently, delitecfg verifies that the user-supplied location of Delite is, actually, a clone of Delite's github repository. This validation seems to be too restrictive to me, though I haven't been able to come up with something better. When a finer sanity check comes to my mind, I'll implement it. For now we can use it as is or comment it out.

delitec is the first step in the compilation pipeline. It reads the configuration file, verifies the settings it uses, automatically sets the classpath and runs automatically detected scala-virtualized compiler to initially build the app.

delites is used to trigger staging. This script uses the configuration produced by delitecfg, automatically sets the classpath and runs automatically detected scala-virtualized interpreter to generate code from a runner provided by the user.

Finally, delite invokes the runtime to execute a staged app using a deg file provided by the user. Similarly to the previous two scripts, it reads the configuration and automatically infers stuff like classpath and location of vanilla scala libraries.

All pipeline scripts assume that the corresponding projects have already been built by the user (i.e. that LMS has been compiled and published + Delite and its runtime have been successfully compiled). I think, it's reasonable to decouple the compilation of Delite itself (which reduces to simply invoking SBT) from the usage of Delite.

The scripts use bash 4.x, though only the first one (delitecfg) is complex and will need to be rewritten for Windows - the others 3 scripts (delite compilation pipeline) can be easily ported. In fact, I plan to reimplement delitecfg using scripting capabilities of Scala itself (let's not burden Windows users with requiring Python or Perl and just use what we have).

[xeno-by]

Here is the RC version of scripts that work both for Linux (I've tested on Ubuntu 11.04 x64) and Windows (I've tested on Windows 7 x64): delitecfg, delitecfg.bat, delitec, delitec.bat, delites, delites.bat, delite, delite.bat, sbt, sbt.bat, sbt.jar. upd. These links no longer work, since I've pushed the scripts to this repo at https://github.com/stanford-ppl/Delite/tree/develop/scripts.

Scripts require: sh interpreter when on Unix (not bash, since shell scripts in this package are really simple), batch interpreter when on Windows, and any scala interpreter at system's PATH to run delitecfg (I've tested it with 2.8.1 and 2.9.0.1 - both worked finely). Scripts rely on sbt 0.7.4 (to inspect project info), though sbt is bundled along with the scripts, i.e. this dependency produces no additional requirements.

Scripts work correctly even if input filenames contain whitespaces. However they will fail if the path to Delite's home directory contains whitespaces (on Linux the runtime has a problem compiling code generated during staging, on Windows virtualized scala interpreter has troubles with understanding quoted -Ddelite.home.dir).

The usage scenario is as follows: 0) Scripts assume that the user has already built both LMS and Delite (i.e. successfully run "sbt publish-local" for the prior and "sbt compile" for the latter). 1) delitecfg/delitecfg.bat. These scripts ask the user for the home directory of Delite and infer everything else on its own. Generated configuration is saved to user's home directory (~/.delitecfg on Unix, %USERPROFILE%.delitecfg.bat on Windows). 2) delitec/delitec.bat. These scripts compile the app with scala-virtualized using settings generated by delitecfg. 3) delites/delites.bat. These scripts stage the app with scala-virtualized using settings generated by delitecfg. 4) delite/delite.bat. These scripts invoke Delite runtime using user-provided DEG file and optional parameters.

Finally, I haven't been able to test scripts on Mac OS, since neither I nor my friends own a Mac. Maybe, anyone here can give me a hand in this endeavor? Possible failure points might include I/O redirection for running sbt (it gave me a lot of pain on Windows!) and detection of user's home directory (currently I use ${HOME:-LOGDIR} to do that).

[michaelmwu]

i'll try these out when i get the chance!

[xeno-by]

I've just pushed the scripts to this repo at https://github.com/stanford-ppl/Delite/tree/develop/scripts. Please, use them instead of the stuff from pastie/Dropbox that I've linked above.

[xeno-by]

Tested the scripts on Mac OS X 10.6.6. Found some minor bugs and fixed them => closing this issue, since everything works now.