Speed up the Larceny build process

_Reported by: pnkfelix on Fri Sep 14 12:24:08 2007 _ (related tickets: #122 #415 #65 #84 #71)

Some discussion on the [http://scheme-punks.cyber-rush.org/wiki/ ERR5RS] wiki indicated that Larceny builds can take an absurdly long time compared to other Scheme systems.

These links probably won't remain valid forever, but I might as well link directly to the actual complaints
[http://scheme-punks.cyber-rush.org/wiki/index.php?title=Talk:ERR5RS:A_Concrete_and_Conservative_Proposal nmh complaint]
[http://scheme-punks.cyber-rush.org/wiki/index.php?title=User_talk:Nmh nmh discussion of machine speed.]

So Felix is opening this ticket as a point to collect:

Tools for instrumenting the build process
Statistics on how long the build takes

The [http://larceny.ccs.neu.edu/dart Dart] autobuild has timing information each night for how long each of the builds took.

Here's the email (after slight edits) I sent to Will on the matter on September 12th, 2007:

On the Twobit pass times: I gathered numbers on SunOS as well as Linux.

As you know, assembly on SPARC takes little time compared to overall compilation time. On SPARC, 9.1% of compilation time is spent in the assembler.

According to the data I gathered, assembly on IA32 takes a significant amount of time, but it is not fair to claim that it takes 75% of compilation time. On IA32, 28.9% of compilation time is spent in the assembler. 21.8% of compilation time is spent within the Sassy assembler itself; the remaining time can be charged to code I wrote to interface with Sassy.

There is a difference between the SunOS and Linux beyond what assembler they use in their compiler, and this probably accounts for the difference in the nightly builds that you used to derive the 75% figure. Each system needs to compile the corresponding files for each of their assemblers. Maybe that was obvious to you, but it wasn't a cost that I was thinking of when we spoke during our meeting.

The reason that this is relevant is that it takes a noticeably long time for us to compile some of the files for Sassy. ([source:trunk/larceny_src/src/Lib/Sassy/opcodes.scm opcodes.scm] is particularly painful to load). Plus we end up effectively compiling several of these files twice: an in-heap compile when we initially load the files into Larceny, and a second compile to a fasl file when we do the (build-larceny-files) step. So we end up getting charged twice for the time spent compiling any particular file in Sassy.

        Elapsed Time in ms
            IA32        SPARC
-----------------------------------------------------
COMPILE:        211,678         184,772
    PASS1:       89,016 (30%)   110,128 (54%)
    PASS2:        1,909 ( 1%)     3,419 ( 2%)
    PASS3:      112,483 (38%)    58,589 (29%)
    PASS4:        4,802 ( 2%)     6,917 ( 3%)
ASSEMBLE:        85,885 (29%)    18,650 ( 9%)

OVERALL:        297,563     203,422

Here is similar data, but broken down according to whether the time is being spent:

compiling source code while loading Twobit
compiling the standard heap
compiling the Larceny fasl files.

                         IA32         SPARC
LOAD-TWOBIT
 * PASS1               41,832       108,975           
 * PASS2                  907         1,318
 * PASS3               78,312        19,321
 * PASS4                2,414         3,292
 * ASSEMBLE            45,021         4,365
BUILD-HEAP
 * PASS1               47,328        61,905
 * PASS2                  964         2,048
 * PASS3               20,454        36,263
 * PASS4                1,969         3,719
 * ASSEMBLE            28,681        13,961
BUILD-FASL
 * PASS1               41,733        49,354
 * PASS2                1,020         1,483
 * PASS3               85,756        22,823
 * PASS4                2,767         3,421
 * ASSEMBLE            56,262         4,958

pnkfelix / larceny-test

Speed up the Larceny build process #472