CLEANUP should produce DFASL or LCOM consistent with what's already there and/or what was previously loaded

[rmkaplan]

There are now 2 kinds of compiled files in Medley, with extensions DFASL and LCOM. DFASLs are produced by the common lisp compiler, LCOMs are produced by the byte compiler.

Some files have both DFASL and LCOM versions, that may be compiled on different dates and maybe even on different source versions. The one that gets loaded is likely determined by the order of extensions on a particular list. So this is not a good situation: for each source file there should be only one type of compiled file.

But also, this raises the question, why does the system as a whole have 2 different compilers and 2 different compile-file formats?

The LCOM files have the advantage that incremental loading of single compiled functions via LOADFNS is possible, and that is very useful in debugging contexts. Incremental loading is not possible from DFASL files.

DFASL files presumably have the advantage of faster loading, but on modern machines load-time is in the noise.

So: does anybody know whether the common-lisp compiler and DFASL files have other advantages that would be lost if we gradually deprecated them and reverted everything to LCOM? Are there constructs that only the common lisp compiler can compile, or does it have better or different optimizations than are provided by the byte compiler?

[nbriggs]

At least one DFASL shows up in the (instructions for the) MAKEINIT process. At this stage of our understanding I wouldn't count on it being able to use an LCOM interchangeably.

[masinter]

they emit different code; we should run the benchmarks with different compilers

[rmkaplan]

As part of the cleanup effort, if a symbolic file had compiled files of both types, I eliminated the one that was not specified by its compiler-type property. There should not be files with both lcom and dfasl now, and we should avoid creating new confusions.

I think that the various compiler functions (TCOMPL, compile-file) should be fixed to cause an error or at least a warning that requires confirmation if they are asked to compile a file with the wrong property. This is the sort of check that SAMEDIR does to prevent files from accidentally ending up in the wrong place. I think Larry didn't think that was a good idea, and so it is still hanging.

But without check such checks, manually calling the wrong compiler (as opposed to letting cleanup figure it out) will create the confused situation.

This is a separate issue from deciding whether one compiler produces better code than the other.

[masinter]

My only concern is that SAMEDIR might interfere with a working style where users want to treat the distribution as read-only, and store updates in their project directory. I wasn't sure about the implications.

As for DCOM and DFASL, a warning would be fine, but warnings tend to get lost.

[rmkaplan]

On SAMEDIR, seems odd to me that someone would have a separate “project directory” for files that are otherwise cloned from git, especially if they are working on medley system files. What’s the point of all the cloning if there is a working style where people don’t mind if files get randomly sprayed by default and without warning to whatever directory they happened to be connected to. That’s how things get out of sync.

If the issue is that they might want to back out some experimental changes, well, that’s what our local versions are for and also why they have to explicitly add/commit etc to update the repository.

I changed SAMEDIR some time ago, before this became a community effort, to have a timeout after which it would default (“oops”) to the file’s original directory…and switch back after the makefile so you stay where you thought you were. Those are parameters that can be set. But I didn’t have a SAMEDIR tedit file at the time, so I didn’t update the scanned documentation,

On Dec 24, 2020, at 12:41 PM, Larry Masinter notifications@github.com wrote:

My only concern is that SAMEDIR might interfere with a working style where users want to treat the distribution as read-only, and store updates in their project directory. I wasn't sure about the implications.

As for DCOM and DFASL, a warning would be fine, but warnings tend to get lost.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/16#issuecomment-751008128, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJIOON27E4LUJSK2KQTSWORQBANCNFSM4QJ22L6Q.

[masinter]

I've been remiss about not gathering together documentation about changes we've made to the default full loadup. I think we need to do that I'm not sure how / where to do that, though. Updates to the Medley 2 user's guide? Google docs or GitHub Wiki?

[masinter]

I thought this was fixed, but the other day I did a ED(MEDLEY-INIT-VARS) which did a LOADFROM(LOADUP-LISP PROP) when I did a CLEANUP it made a DFASL rather than LCOM

Any ideas? @rmkaplan

[rmkaplan]

Something else that seems to be confused, it was noted but never addressed.

It appears that the choice of compiler is driven off the FILETYPE property, which only some files actually have.

There also seem to be 4 different compile modes: CL:COMPILE-FILE, FAKE-COMPILE-FILE, TCOMPL/RECOMPILE, and BCOMPL/BRECOMPILE.

I think the T/R and B/BR are the same if the file doesn’t have any block declarations. That distinction should be eliminated.

The CL:COMPILE-FILE produces DFASLs, the FAKE-COMPILE-FILE appears to be able to compile files with commonlisp constructions (like FUNCTIONS), but it produces LCOMs.

The properties are not always specified the way they should be. There needs to be a predicate on the FILECOMS that decides what kind of compiler it needs to use. I don’t understand whether the COMPILE-FILE and FAKE-COMPILE-FILE can compile all the same stuff, and whether one of them produces better code than the other. If they are equivalent, then why don’t we stop using the COMPILE-FILE? For one thing, I don’t think you can LOADFNS from DFASLs.

This is all beyond my understanding.

On Aug 2, 2021, at 5:00 PM, Larry Masinter @.***> wrote:

I thought this was fixed, but the other day I did a ED(MEDLEY-INIT-VARS) which did a LOADFROM(LOADUP-LISP PROP) when I did a CLEANUP it made a DFASL rather than LCOM

Any ideas? @rmkaplan https://github.com/rmkaplan — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/16#issuecomment-891409301, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJN776QZOOURCTMAYSTT24WSTANCNFSM4QJ22L6Q.

[masinter]

check out internal/library/MULTI-COMPILE.

[rmkaplan]

Interesting.

It doesn’t actually find source files, it only finds source files per se, it only finds source files that already have compiled files. Should use LISPSOURCEFILEP as the predicate.

Maybe the core piece should be taken out of this and put somewhere else in the loadup (filepkg?) Other stuff in here seems to be historical, but the core piece is what I have been looking for: do the right thing.

I would extend it, perhaps, so that you could also give a list of functions to be recompiled, and then it chooses brecompile/recompile in the cases where now the compiler-type is bcompl/tcompl. You would end up in the same place, presumably, but the typescript would be meaningful with respect to just the functions that you are updating (e.g. that you didn’t edit but got from a who-calls or who-uses masterscope query).

On Sep 19, 2021, at 12:38 PM, Larry Masinter @.***> wrote:

check out internal/library/MULTI-COMPILE.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/16#issuecomment-922525221, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJPWMT32BCH73DBB74TUCY33FANCNFSM4QJ22L6Q.

[masinter]

I want to get to the place where we can compile everything, starting from 'starter.sysout'.

we're not far from being able to do that. Then we can .gitignore .LCOM .DFASL and instead ship them as part of a release

[masinter]

both il:load and cl:load can load either. Why don't we settle on a single file extension (.dfasl? .ilc? whatever) and reduce the complexity of loading?

[nbriggs]

They're still different formats, and while il:load and cl:load can both load both formats, there are operations you can do on LCOM format files that you can't do on DFASL format files -- so getting rid of the extension means that a person would now have to look in the file to see whether the contents were going to be suitable to do the operation they want... and that seems way more involved than just looking at the extension. I vote we have separate extensions for separate formats.

[rmkaplan]

The distinction at load-time is not between dfasl and lcom, it’s between loadable-compiled files and loadable-source files. So maybe what we want is a convenient new entry point (CLOAD X) that you use to load whatever compiled file there is that corresponds to source file X…and gives an error if there is both a dfasl and an lcom. In that case, something went wrong, one may be out of date, etc.

On Sep 25, 2021, at 4:24 PM, Larry Masinter @.***> wrote:

both il:load and cl:load can load either. Why don't we settle on a single file extension (.dfasl? .ilc? whatever) and reduce the complexity of loading?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/16#issuecomment-927197582, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJP23YBWVGP354C5RY3UDZK4PANCNFSM4QJ22L6Q.

[masinter]

DOFILESLOAD (called by FILESLOAD, which is what implements the FILES Interlisp filepackagecommand) is what we use to load the appropriate LCOM or DFASL file. I think we've eliminated the files that had both LCOM and DFASL files. I think all the files in the loadups have FILETYPE properties. What else do we need to do to close this issue?

[masinter]

reviewing old issues... DFASLs have the feature (also with FAKE-COMPILE-FILE) that top-level S-expressions are run compiled... eliminating some awkward behavior with MACROs and records not loaded in order. I am not sure but I think there are also loadup issues with having .DFASLs too early in the loadup. I know there are optimizations in DFASLs that aren't in the interlisp compiler, but I fear it's also vice versa.