Want simplified Medley startup

[masinter]

if there's an issue on this, I can't find it.

The current situation with "run-medley" as one entry point and "lde" "ldex" "ldeinit" as runtime binaries with different options parameters, primarily because of the perception that the C code was more difficult to update, had unknown compatibility constraints. The idea (not quite a plan) was that we would settle on a better interface and set up a new entry point.

For a transition transition period, we could keep on creating lde / ldex / ldeinit and leave run-medley, but make the new binary also.

Desired properties:

• one executable, not 3 (possible?)
• avoid using environment variables to communicate (LDESRCESYSOUT etc)
• startup options expressed as "--param" or "-p" short form

Call it 'medley'?

Related issues that might be affected:

PR #928 to be location and update status of lisp.vmem -- default to a /tmp/medleyNNN.vm ... a while back there was a change to the Lisp code to do versioned sysouts (first done in PR #277, reverted in PR #291, and then reubmitted in PR #296) done in Lisp by writing out the sysout as name.sysout-temp.

[nbriggs]

Some things are easy, some not so much, unless we want to rewrite a lot of code.

There is one executable that a user should execute directly (lde), and behind the scenes, one executable for each distinct windowing system or piece of display hardware (ldex, ldesdl, ldecgsix, etc.). The different display systems have conflicting include files and symbols, so it's not possible to compile both in the same piece of code. ldeinit is a special-purpose piece of code, which nobody should be invoking independently outside of building new system images from source. There's also ldeether to think about -- it's a privileged helper process so that you're not running the whole lisp environment as root when you need (on some systems) to be root in order to be able to open the raw ethernet device (or have libpcap open it for you).

/tmp is a terrible place to put a virtual memory file because it is often cleared when a system is rebooted.

We can certainly make the sysout and vmem file names command line parameters. We can also relatively easily switch to the GNU style switches with --param and -p. We do have to account for the fact that there are potentially different switches for each display subsystem, and some display subsystems (like X11) have their own way of passing information in such as the resource database, so not everything can be put on the command line.

Getting rid of the LDESRCESYSOUT/LDEDESTSYSOUT is fine with me, we just have to get the file names we'll need onto the command line.

The design I'm thinking about is more about being able to easily specify the right behavior when you (1) want to start a fresh system from some sysout, but may want to specify whether the VM it's to use should overwrite an existing one (which one?) or be unique, and (2) want to restart a particular vm representing a session from which you logged out (or were otherwise disconnected) and how do you know which one that was...

[orcmid]

I know this is far off the mark here, but a related question occurs for me.

I understand that there needs to be a memory store for Interlisp to deal with garbage collection, referencing within the store, etc. But I wonder if there should be greater exploitation of the fact that the host operating system and the hardware may be operating with virtual memory already.

My suspicion is that wanting to store an instance to a file system (a Sysout?) and be able to resume it from there is a factor in all of this and what I am wondering about is simply unspeakable.

[nbriggs]

It has been that way since day one… lisp on Maiko has never handled its own paging, and just has the underlying o/s deal with it after allocating a chunk of address space… originally with Malloy, now with map of anon memory.

Sent from my iPhone

On Sep 11, 2022, at 10:00, Dennis E. Hamilton @.***> wrote:

 I know this is far off the mark here, but a related question occurs for me.

I understand that there needs to be a memory store for Interlisp to deal with garbage collection, referencing within the store, etc. But I wonder if there should be greater exploitation of the fact that the host operating system and the hardware may be operating with virtual memory already.

My suspicion is that wanting to store an instance to a file system (a Sysout?) and be able to resume it from there is a factor in all of this and what I am wondering about is simply unspeakable.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[masinter]

well, maybe day two? If 'day one' was the D-machine era? There was a time I think when there were plans? support? for running the same sysout on D-machine and also on Sun. I'm not sure of the events.....The loadup procedure still goes through the useless process of determining which VM pages had (compiled Lisp) code that was needed to handle a page fault. Some of the LLFAULT code runs renamed and I haven't been eager to tackle that particular mess.

The simplest from the system point of view would be

• msiko: no rrfault place to stave your session
• the 'site' init file can set a place, and users can override. If you give it a {DSK} name it will save a new version. If you give a {UIX} name it will over-write.

The /tmp directory seems just fine for online guests

[nbriggs]

From day 1 of Maiko. You always could (and still can) take a D-machine sysout and run it on Maiko's lde. So the sysout is going to have the D-machine page fault handler code (written in Lisp) in it, and as Maiko brings up the running image it has to make sure that code isn't going to run. I don't believe we could take images that were run on a Sun and bring them back to a D-machine.

[nbriggs]

@masinter -- lde needs to know where it's loading from before Lisp starts running, and, if we're going to make it so that a broken X connection causes it to save the current vm state then it needs to know whether it can store back to what it started from (e.g., if it was resuming a previous session) or it has to create a new saved image with a name distinct from what it loaded from. The site init may not have run to completion before lde is called upon to save the state, so waiting until then to get a vm filename is problematic. Setting LDEDESTSYSOUT is currently how you override the default "lisp.virtualmem" name for a SAVEVM. LDESRC[E]SYSOUT is how you tell it where to start from if it is NOT to start from "lisp.virtualmem", but it will never write back over the LDESRCSYSOUT. But I think I said all this earlier.

I think Frank proposed specifying a tag to name a sessions (rather than a timestamp) when you start a new session or resume an old one -- that's currently trivially implementable by setting the LDEDESTSYSOUT name based on the tag (I think he did that...?, I don't have that thread in front of me)

[fghalasz]

@masinter >The /tmp directory seems just fine for online guests

No need to treat online guests as any different from registered online users. Online guests do not have a permanent docker volume mounted as do registered users. That means their entire file system is contained within the Medley docker container, which is blown away as soon as they log out. Hence all of their modifications to the file system are effectively "tmp".

For online, the lisp.virtualmem is always stored in /home/medley/il/vmem/lisp.virtualmem. This is setup in the run-online-medley script (which is the docker RUN script). run-online-medley sets up LDEDESTSYSOUT with the above path (among many other things) and then calls run-medley.

[masinter]

This was discussed further in a PR discussion: https://github.com/Interlisp/medley/pull/1358#issuecomment-1771949122 and messages following.

I'd like the ability to call Medley from Medley, in order to move some of our automation into Lisp. Using environment variables to communicate has the problem that there is no way from LIsp to set environment variables .... perhaps a kind of "tail recursive" call where a lisp image can start over from a savedvm without taking the screen down ....

[nbriggs]

It should be easy to start a new image - essentially doing a logout with a new image name - though there are details to be worked out like what effect the lingering environment variables for things like the destination image should have. Keeping the same X/SDL window is not trivial, but shouldn't be too difficult.

[masinter]

I'm wondering how many of the environment variables used in run-medley could be eliminated and replaced with inputs -- to avoid the unwanted inheritance. I wonder if we could remove the dependence on LDEDESTwhatever by just making SAVEVM and LOGOUT to use the parameter (if supplied).

[nbriggs]

There don't need to be any explicitly set environment variables - but it does need the "place to store the VM" at times other than when you're doing a LOGOUT or an explicit SAVEVM. Those can certainly be command line (or whatever) parameters to lde, and the "image to restart with" can be an optional string argument to LOGOUT. I wouldn't add an option to SAVEVM - that should still be saving to the "default" (or as specified on the command line) image destination. For sure you don't want the destination image to default to the startup image.

[nbriggs]

The environment variables, set externally (not including ones used by lde to communicate with ldex/ldesdl) that the code currently depends on are:

ldeether.c:    "LDEETHERDEV" (guides choice of raw ethernet device)
main.c: "LDESRCESYSOUT" or  "LDESOURCESYSOUT" (the file containing the image to start from)
main.c: "HOME" (not generally set by user; user's home directory)
timer.c: "LDEFILETIMEOUT" (time in seconds to interrupt file system ops; default 10s)
unixfork.c: "SHELL" (not generally set by user; the shell that will be invoked in "chat to shell")
vmemsave.c: "LDEDESTSYSOUT" (the file where the virtual memory image is written; default lisp.virtualmem)
xrdopt.c: "DISPLAY" (often set by system (e.g. by ssh -X), can be overridden by user)

internally it uses these environment variables to communicate between processes, which need to be reset when a new lde/lde\<display> pair is started:

unixcomm.c: "LDEPIPEIN"
unixcomm.c: "LDEPIPEOUT"
unixcomm.c: "LDESTARTTIME"
unixcomm.c: "LDEUNIXPID"

[masinter]

There don't need to be any explicitly set environment variables - but it does need the "place to store the VM" at times other than when you're doing a LOGOUT or an explicit SAVEVM.

Are there any places that "store the VM" other than LOGOUT, SAVEVM, SYSOUT and MAKESYS?

I was pretty sure there weren't.

[nbriggs]

The other one (that isn't currently functional, but should be) is in the maiko code when the Lisp is about to shut down because the window system (X11) has crashed, or you close the window. We should still have the ability to SAVEVM periodically in the background (i.e., an implicit SAVEVM, where you're not going to be interacting to provide a filename).

[nbriggs]

These all map down to one place in the Maiko code - where it has to construct a system image out of all the bits it has (machine state and storage) and write the image to a file (along with the FilePage to VirtualPage mapping table, etc. etc.)

[masinter]

@fghalasz this isn't our first rodeo on this topic.