Modules can be re-compiled multiple times

[apblack]

This is the inverse of #230. If there is an out-of date .js file on GRACE_MODULE_PATH, here is what can happen.

black@ada:/pkgs/minigrace/source$ j2/minigrace-js js/tests/t222_import_test.grace --verbose 50
minigrace: t222_import_test: 0.45 (+0.45): starting compilation
minigrace: t222_import_test: 0.46 (+0.01): lexing.
minigrace: t222_import_test: 0.49 (+0.03): parsing.
minigrace: t222_import_test: 0.54 (+0.05): checking dialect standardGrace used by module t222_import_test
minigrace: t222_import_test: 0.54 (+0): checking module "standardGrace"
minigrace: t222_import_test: 0.55 (+0.01): found module "standardGrace" in /pkgs/minigrace/lib/grace/modules/./standardGrace.js
minigrace: t222_import_test: 0.66 (+0.11): checking module "collectionsPrelude"
minigrace: t222_import_test: 0.66 (+0): found module "collectionsPrelude" in /pkgs/minigrace/lib/grace/modules/./collectionsPrelude.js
minigrace: t222_import_test: 0.79 (+0.13): executing sub-compile "/pkgs/minigrace/source/j2/minigrace-js" --verbose 50 --gracelib /pkgs/minigrace/source/j2/ --target js --make "/pkgs/minigrace/source/collectionsPrelude.grace"
minigrace: collectionsPrelude: 0.45 (+0.45): starting compilation
minigrace: collectionsPrelude: 0.65 (+0.2): lexing.
minigrace: collectionsPrelude: 2.85 (+2.2): parsing.
minigrace: collectionsPrelude: 8.47 (+5.62): rewriting tree.
minigrace: collectionsPrelude: 10.76 (+2.29): symbol tables built.
minigrace: collectionsPrelude: 16.93 (+6.17): generating JavaScript code.
minigrace: collectionsPrelude: 23.89 (+6.96): done.
minigrace: t222_import_test: 25.63 (+24.84): no need to load dialect "standardGrace": it does not define `thisDialect`
minigrace: t222_import_test: 25.64 (+0.01): rewriting tree.
minigrace: t222_import_test: 25.74 (+0.1): checking module "t222_imported"
minigrace: t222_import_test: 25.74 (+0): about to compile module "t222_imported"
minigrace: t222_import_test: 25.74 (+0): executing sub-compile "/pkgs/minigrace/source/j2/minigrace-js" --verbose 50 --gracelib /pkgs/minigrace/source/j2/ --target js --make "/pkgs/minigrace/source/js/tests/subtest/t222_imported.grace"
minigrace: t222_imported: 0.45 (+0.45): starting compilation
minigrace: t222_imported: 0.46 (+0.01): lexing.
minigrace: t222_imported: 0.47 (+0.01): parsing.
minigrace: t222_imported: 0.52 (+0.05): checking dialect standardGrace used by module t222_imported
minigrace: t222_imported: 0.52 (+0): checking module "standardGrace"
minigrace: t222_imported: 0.53 (+0.01): found module "standardGrace" in /pkgs/minigrace/lib/grace/modules/./standardGrace.js
minigrace: t222_imported: 0.6 (+0.07): checking module "collectionsPrelude"
minigrace: t222_imported: 0.6 (+0): found module "collectionsPrelude" in /pkgs/minigrace/lib/grace/modules/./collectionsPrelude.js
minigrace: t222_imported: 0.67 (+0.07): executing sub-compile "/pkgs/minigrace/source/j2/minigrace-js" --verbose 50 --gracelib /pkgs/minigrace/source/j2/ --target js --make "/pkgs/minigrace/source/collectionsPrelude.grace"
minigrace: collectionsPrelude: 0.45 (+0.45): starting compilation
minigrace: collectionsPrelude: 0.63 (+0.18): lexing.
minigrace: collectionsPrelude: 2.69 (+2.06): parsing.
minigrace: collectionsPrelude: 7.97 (+5.28): rewriting tree.
minigrace: collectionsPrelude: 10.22 (+2.25): symbol tables built.
minigrace: collectionsPrelude: 16.17 (+5.95): generating JavaScript code.
minigrace: collectionsPrelude: 22.9 (+6.73): done.
minigrace: t222_imported: 24.44 (+23.77): no need to load dialect "standardGrace": it does not define `thisDialect`
minigrace: t222_imported: 24.44 (+0): rewriting tree.
minigrace: t222_imported: 24.54 (+0.1): symbol tables built.
minigrace: t222_imported: 24.57 (+0.03): generating JavaScript code.
minigrace: t222_imported: 24.64 (+0.07): done.
minigrace: t222_import_test: 51.08 (+25.34): found module "t222_imported" in /pkgs/minigrace/source/js/tests/subtest/t222_imported.js
minigrace: t222_import_test: 51.13 (+0.05): symbol tables built.
minigrace: t222_import_test: 51.16 (+0.03): generating JavaScript code.
minigrace: t222_import_test: 51.27 (+0.11): done.
submodule initialization
submodule has been imported
black@ada:/pkgs/minigrace/source$

Notice that collectionsPrelude.grace is re-compiled twice. An out-of-date version is found in /pkgs/minigrace/lib/grace/modules/./collectionsPrelude.js, which is therefore recompiled, creating a new binary in /pkgs/minigrace/source/, the same directory in which the source code was found.

ls -l /pkgs/minigrace/source/collectionsPrelude.*
-rw------- 1 black them  57960 Sep  9 09:07 /pkgs/minigrace/source/collectionsPrelude.grace
-rw------- 1 black them 840620 Sep  9 10:49 /pkgs/minigrace/source/collectionsPrelude.js

However, the next time that collectionsPrelude is checked, the same thing happens again; the freshly compiled version is not found. This is because the current GRACE_MODULE_PATH

/pkgs/minigrace/lib/grace/modules

does not contain the current directory.

Why, then, is the source code found in the current directory? There must be some inconsistency as to if, and where, the current directory is added to the search path.

[apblack]

I have also seen this happening in an interesting situation. I created a directory of files using a zip file exported from the web IDE. Because of a timezone issue, all of the source files were timestamped tomorrow. This meant that even after a file was compiled, the object file was not newer than the source file. Thus, it was compiled again — several times (once for each import) — for a single top-level compilation.

Unless one turns on --verbose, it seems like the compiler is just running very slowly.

[apblack]

Here is a concrete scenario where this issue arrises.

Two modules being compiled import ast. Source file ast.grace is in directory .., while object file ast.js is in /usr/local/lib/grace/modules. Source is newer than object, so source is recompiled. There is no --dir flag, so new object file also goes in directory ..

The snag: GRACE_MODULE_PATH=/usr/local/lib/grace/modules/:../js:..:../js/tests

Hence, next time the sources are compiled, the out-of-date object file is again found, since it is on GRACE_MODULE_PATH ahead of the up-to-date one.

I think that the way this ought to work is that we look for the first .grace file on the GRACE_MODULE_PATH. That is the file we should be using, because we have said that Grace programs are represented as source, and several other implementations are interpreters. If we can't find a compiled version of that file that corresponds to that source, we have to compiler it again. We look for the compiled version on the same path, but we should not look for the first compiled version on the path — we should look for a version that has the same SHA as the source.

This means computing the SHA and putting it in the name of the compiled file, or at least inside the file.

An alternative solution is to change the way that the object-file is sought. Instead of looking for the first file on the path, and checking that it is newer than the source, we could instead look for the first file on the path that is newer than the source, simply ignoring all older files (which can't possibly help us). This might give us the wrong file, but it won't cause multiple compilations (assuming that we add any --dir argument to the path).

[apblack]

This is resolved by commit 0cf0402, which used a SHA-256 checksum to find an appropriate compiled code file, ignoring file times.

See also issue #293, which I hope is resolved by the same commit.