Get rid of LOADUPDIRECTORIES, after loadup ?

[rmkaplan]

LOADUPDIRECTORIES seems to be used in the MAKEINIT, but it persists in the MAKESYS. I don't think it is necessary there, and it is now a little confusing to have both LOADUPDIRECTORIES and LOADUPSDIRECTORIES, where the sysouts, exports, whereis are found

[masinter]

There is a ton f junk in the loadup process left over from the days when doing a loadup was an all day manual process. If I were to fix this (which I don't think is worth the trouble it would cause) I would eliminate LOADUPDIRECTORIES in favor of just setting DIRECTORIES (which is all LOADUPDIREFTORIES is used for) and change the latter to LOADUPSDIRECTORY (is there ever more than one?) or just use (MEDOEYDIR "loadups")

[fghalasz]

LOADUPDIRECTORIES is used in MAKEINIT and LOADUP-LISP which are both completed before the MAKESYS for lisp.sysout in loadup-lisp-from-mid.sh.

Couldn't we just unbind LOADUPDIRECTORIES immediately before the call to MAKESYS in the "CM file" included in loadup-lisp-from-mid.sh c?

Unbinding would just be (SETOPVAL 'LOADUPDIRECTORIES 'NOBIND), correct?

[nbriggs]

Yuck. When we made sysouts at PARC they went into "basics" not "loadups" (and we didn't find the need to have a BASICSDIRECTORIES). The EXPORTS.ALL and the WHEREIS.HASH used to go into the released library directory, where they'd be found from just plain DIRECTORIES, so the place you had the sysouts was irrelevant to a running lisp system. We seem to be tying ourselves into knots because we made a bad choice... which we can choose to undo rather than hack around.

[fghalasz]

Two separate issues here:

1. Unbinding LOADUPDIRECTORIES before lisp.sysout MAKESYS because it is used only in the loadup process. Good idea IMHO even if there is no confusion with LOADUPSDIRECTORIES.

2. The need for LOADUPSDIRECTORIES -- Seems to me putting the products of the loadup process (e.g., WHERSIS.HASH) into library doesn't fit the "meaning" of library. That being said we could easily add the loadups subdirectory to DIRECTORIES and get rid of LOADUPSDIRECTORIES altogether. As far as I can see LOADUPSDIRECTORIES is not used in any code other than to set it. So seems like a relatively easy change.

[rmkaplan]

Should we also get rid of LISPUSERSDIRECTORIES, just have everything (except for fonts) in one DIRECTORIES list?

The current segregation is a kind of heavy-handed way of dealing with priorities of files with the same name in different directories (disaster in itself) and of speeding lookups with potentially shorter search paths (pointless, unless there are still references to pup hosts hanging around).

But there are probably lots of occurrences of (FROM LISPUSERS) that would have to be culled out in the long run, but in the short run DOFILESLOAD could simply treat (FROM LISPUSERS) or (FROM LOADUPS) as equivalent to (FROM DIRECTORIES) (= the default).

This would simplify just a bit what a new user has to think about in setting up a personal init file.

On Oct 19, 2023, at 7:31 PM, Frank Halasz @.***> wrote:

Two separate issues here:

Unbinding LOADUPDIRECTORIES before lisp.sysout MAKESYS because it is used only in the loadup process. Good idea IMHO even if there is no confusion with LOADUPSDIRECTORIES.

The need for LOADUPSDIRECTORIES -- Seems to me putting the products of the loadup process (e.g., WHERSIS.HASH) into library doesn't fit the "meaning" of library. That being said we could easily add the loadups subdirectory to DIRECTORIES and get rid of LOADUPSDIRECTORIES altogether. As far as I can see LOADUPSDIRECTORIES is not used in any code other than to set it. So seems like a relatively easy change.

— Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/1357#issuecomment-1771980652, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJMQ6SCEOJOWCTSM5W3YAHO75AVCNFSM6AAAAAA6GKX4RKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZRHE4DANRVGI. You are receiving this because you authored the thread.

[nbriggs]

The WHEREIS.HASH should represent the location of all the indexed items that are in a particular snapshot of the library, which went into a particular set of sysouts. If anything in the library changed then the WHEREIS.HASH should change too. I wouldn't include the LISPUSERS packages in the library WHEREIS.HASH.

[nbriggs]

The same goes for the EXPORTS.ALL - it should match a collection of library packages -- and the same goes for the sources - WHEREIS.HASH and EXPORTS.ALL should cover both library and sources - at a particular snapshot from which we built a lisp.sysout and full.sysout (and notescards.sysout and loops.sysout). If you build a new sysout of your own then you should build new WHEREIS.HASH and EXPORTS.ALL and store them in your own location, and make sure that the correct files get located in the startup.

[nbriggs]

I'm vehemently opposed to "fixing" DOFILESLOAD to cope with special cases of (FROM LISPUSERS) (or any other particular names) - it should be uniform, or it's too hard to explain to new users what it's doing, and adds code we have to maintain for no real value. If (FROM TRILLIUM) looks in TRILLIUMDIRECTORIES then (FROM LISPUSERS) should look in LISPUSERSDIRECTORIES.

If we're going to unbind LOADUPDIRECTORIES because it's a good idea, should we also unbind all other variables that are just remnants of the sysout building process because that's just as good an idea?

[masinter]

I closed the issue as unplanned .....

[rmkaplan]

I want the WHEREIS.HASH to index all the items in a given distribution, whether loaded or not in one or more of the distributed sysouts. And a single entry WHEREIS to find the location(s) of everything I might ask about.

We could build different whereis databases for the directories source-library-lispusers, if there is some value in keeping them separate and simulating the union at look-up time. But I don’t see the point of that: I can’t think of a time when I wanted to ask separately whether a function is somewhere in lispusers vs sources vs library. Given only the name, I want to know what files contain it.

EXPORTS.ALL may be a little different, maybe it should (or does) scan only sources/ files. For example, I’m creating a separate TEDIT-EXPORTS.ALL for Tedit, used just to share definitions among the different Tedit files. I load it when you want to work on or compile just those files. Not needed to work on more fundamental system files, but useful to load if you are implementing an application (Notecards? LFG? DOC-OBJECTS?) that integrates more tightly with Tedit internals.

On Oct 19, 2023, at 8:40 PM, Nick Briggs @.***> wrote:

The WHEREIS.HASH should represent the location of all the indexed items that are in a particular snapshot of the library, which went into a particular set of sysouts. If anything in the library changed then the WHEREIS.HASH should change too. I wouldn't include the LISPUSERS packages in the library WHEREIS.HASH.

— Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/1357#issuecomment-1772028606, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJK5QXOA4OJT5G3PDA3YAHXD5AVCNFSM6AAAAAA6GKX4RKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZSGAZDQNRQGY. You are receiving this because you authored the thread.

[nbriggs]

I want the WHEREIS.HASH to index all the items in a given distribution, whether loaded or not in one or more of the distributed sysouts. And a single entry WHEREIS to find the location(s) of everything I might ask about.

Where I said library I should have said library+sources. I think that LISPUSERS may be different - more likely to change - but sure, it could be incorporated as whatever was current when the "release" was made. I believe what drives WHEREIS lookups is a list of hash files because you may want to add your own things that aren't part of the released system.

I think I agree with what you're saying about EXPORTS.ALL - but I'm not sure whether the boundary for EXPORTS.ALL should be just sources, or sources + (library?) packages (subsystems) that are complicated enough to need their own exports, such as TEdit, or Lafite, or Notecards, or Sketch, or... or should they have their own EXPORTS. files (and should they be automatically loaded when you load EXPORTS.ALL?)

[fghalasz]

If we're going to unbind LOADUPDIRECTORIES because it's a good idea, should we also unbind all other variables that are just remnants of the sysout building process because that's just as good an idea?

Yes we should unbind everything used exclusively during and for the loadup process. But its a big and a fairly low priority task to identify everything that should be so unbound. Why not start by doing the task incrementally as specific things (e.g., LOADUPDIRECTORIES) come up? Its easy to do and easy to update incrementally as part of the loadup scripts.

[rmkaplan]

On Oct 19, 2023, at 10:54 PM, Nick Briggs @.***> wrote: Where I said library I should have said library+sources. I think that LISPUSERS may be different - more likely to change - but sure, it could be incorporated as whatever was current when the "release" was made. I believe what drives WHEREIS lookups is a list of hash files because you may want to add your own things that aren't part of the released system.

I don’t think “more likely to change” is the issue, the relased database should always reflect the current state of affairs. Frank fixed it so that the loadup-full script doesn’t automatically go through the slower process of creating the database each time an experimental loadup is made, but the loadup-all script that is used in releases should always scan everything. (Note, for example, that the readimageobject function uses the whereis database as a last-straw heuristic to find the getfn for an image object when you open a Tedit file, that could be in Lispusers.)

I think I agree with what you're saying about EXPORTS.ALL - but I'm not sure whether the boundary for EXPORTS.ALL should be just sources, or sources + (library?) packages (subsystems) that are complicated enough to need their own exports, such as TEdit, or Lafite, or Notecards, or Sketch, or... or should they have their own EXPORTS. files (and should they be automatically loaded when you load EXPORTS.ALL?)

Could be sources+library, but then EXPORTS.ALL would always have Tedit stuff embedded within it, and things could get out of step. I have it set up now so that the Tedit file “exports” a command to load the generic EXPORTS.ALL (I think its (FILES (FROM LOADUPS)…) since TEDIT really does access all sorts of low-level stream, string, and arithmetic macros. For me it feels better to have each of the individual subsystems in control of their own stuff, and explicitly inherit from others.