moving Medley files into standard places when fitting into Unix distros

[masinter]

Converted back to an Issue for more focused discussion

Discussed in https://github.com/Interlisp/medley/discussions/77

^{Originally posted by **blakemcbride** December 9, 2020} # Path Proposal for Medley Revision History ---------------------- Blake McBride 2020-12-09 Original proposal ------------------------------ Problem ----------- Special configuration options notwithstanding, Medley defaults to storing its lisp.virtualmem, SYSOUT files, and makefiles in the user's home directory. This surely made a lot of sense in the days when machines were smaller and more single-purpose. However, this is a problem with modern machines because: 1. It clutters up a user's home directory. 2. When working on multiple projects, the various lisp.virtualmem, SYSOUT, and makefiles get intermixed. Proposal ------------- 1. Medley's executable programs and scripts should be located in the /usr/local/bin directory. 2. Store other common Medley files (such as standard images, system source code, documentation files, etc. in the /usr/local/share/medley directory. /usr/local is called the Medley Home directory. 3. The directory the user was in when Medley is started is called the "Project Directory". 4. By default, Medley accesses the lisp.virtualmem, SYSOUT files, and makefiles in the Project Directory. It may also have pointers to and read files in the Medley Home directory. 5. If there is no lisp.virtualmem file in the Project Directory, Medley will load a standard SYSOUT located under the Medley Home. 6. Under normal circumstances, Medley does not access any files outside of the Medley Home directory or the Project Directory. 7. Unless explicitly changed, Medley will treat the Project Directory as its home or default path. 8. Users are not allowed to write to the Medley Home. 9. The medley build process should have a make target "install" that sets up and installs the Medley Home. Advantages: -------------------------------------- 1. The system will be laid out in a form well understood by the community. 2. The system will be laid out in a practical and clean way that is cooperative with other applications on the system. 3. The system will best support multiple projects as well as multiple users on the same system. Potential Issues ---------------------- 1. Users are not allowed to write to the Medley Home. Therefore, some procedures will need to be used to support efforts to modify Medley system files when modifying Medley-specific files. 2. This proposal makes sense for Linux, BSD, and other standard Unix-like distributions. It may need adjustments for the Apple Mac environment.

[masinter]

https://groups.google.com/d/msgid/interlisp/ee96a244-33ab-18df-39b0-2e99552e6328%40lassi.io

emsil discussion

From: Lassi Kortela Sent: Monday, September 20, 2021 12:44 AM To: interlisp@googlegroups.com Subject: Re: Directory layout and OS packaging

The emulator executables would normally live in /usr/local/bin/{lde, ldex, ldeinit},

This is good. For distribution on current operating systems, I recommend changing the name from lde to something more clearly differentiated like maiko or interlisp-maiko, and adding a hyphen between the program name and the display backend (maiko, maiko-x, maiko-init).

There is another software package called lde somewhat widely available (https://repology.org/project/lde/versions).

and the site initialization file (usually SITE-INIT.LISP) was in /usr/local/lde/. Variables set up in the site init file would normally indicate the location of the other components (fonts, library, lispusers)

Also reasonable. I guess most distros nowadays will want /usr/local/share/lde in order to accept it into their package manager.

Check out medley/docs/Sun-Users-Guide.pdf -- when we were releasing into environments which remote mounted their /usr/share from a central file server (remember those days? Disks were small!) /usr/share/lde was the preferred location for everything but the architecture specific executables.

Thanks! I searched the PDF for all mentions of "/usr" and found the following.

The manual suggests these pathnames under each Unix home directory:

~/medley/
~/medley/*.sysout

And these root directories for system-wide Medley files:

/usr/share/lde
/usr/local/lde (for site-specific stuff?)

Under the system-wide root it refers to the following pathnames:

.medleyKey.hostname
fonts/
fonts/display/
fonts/display/chinese/
fonts/display/presentation/
install.sunos*/
install.sunos*/lde
lisplibrary/
lisplibrary/site-init
lispsysouts/
lispsysouts/*.sysout
site-files/
site-files/*.lcom (LDEINIT)
site-init.lisp

Would it be reasonable if we used the following layout?

/usr[/local]/bin/maiko-* (standard files from OS package manager)
/usr[/local]/share/medley/* (standard files from OS package manager)
/usr/local/share/medley/site/* (site-specific files) ~/medley (user's personal files)

In the above notation, /usr[/local] means either /usr or /usr/local depending on the distro. In particular, BSD package managers like to install everything under /usr/local.

The "site" directory is by analogy with GNU Emacs, which uses

/usr/local/share/emacs/site-lisp/*.

There are always a few distros that use an idiosyncratic file system layout, so all pathnames should be configurable in the build script, but I expect the above would work for the vast majority of them

[masinter]

I think we need to support several styles of users -- there is no "one true disk layout". I'd like to retain the style in use currently which was set up for people who are working on the core of the system and with several copies at different states. A separate "install" step would be a pain.

It would be great to have simple "update" workflow for people who want the daily updates of maiko or medley files.

I recently changed run-medley to not look for maiko if lde is on path

There are release scripts in maiko/bin and in medley/scripts (see loadup-and-releash.sh) . Would it be too much to ask how those would change?

Would it be

[lassik]

The current layout of the medley git repo seems reasonable. Could we set the default Medley Home directory to /usr/local/medley, and replicate the layout of the git repo under that directory? Then git clone https://github.com/Interlisp/medley.git /usr/local/medley would work easily, among other things.

[lassik]

Or git clone https://github.com/Interlisp/medley.git ~/medley for people who want to use it from their home directory.

It's not clear whether files should be automatically merged from /usr/local/medley and ~/medley somehow. Maybe not?

[masinter]

there is no "default medley home": different kids of users will want different setups. Right now you can install medley maiko anywhere in your file system or shared drive, Some users will want some advice? If the issue is default defaults, what should be the default if you haven't set anything -- I'd like to see releases for Mac and Windows non-technical, non-unix people.

[masinter]

in Interlisp, you don't really deal with "files" except as something you manipulate within lisp. There are variables usually ending in DIRECTORIES which tell Lisp where to look for files.

In Interlisp, a "file" isn't a file in the OS's file system -- A "file" is what you get when you MAKEFILE or CLEANUP (which gives you two files, one the compiled version of the other). Files contain definitions, definitions define a name to whatever the definer defines --

It's really a different model of how the file system interacts with programming

[masinter]

random issue chooser sent me here. I think this is all post-loadup post-build. Maybe after "release" automation? A separate "install" or "distribute"?

On the rename from 'lde' -- I think 'medley' would be better than 'maiko'. I want to fold 'run-medley' into the C code.

This probably should wait until we get SDL integrated.

We need to automate building for different environments: Mac, Windows, Docker, Linux Docker might have two targets: Local Docker as alternative to WSL and remote "cloud" docker.

[masinter]

re name: we could call the executable to "medley". (over "maiko"). "interlisp?" might interact with #408 The reason for "run-medley" was because it was easier to change than the C code. 'LDE' stood for Lisp Development Environment, I believe.

[nbriggs]

'LDE' stood for Lisp Development Environment, I believe.

I would assume by analogy with XDE, the Xerox Development Environment.

[lassik]

If the Unix shell commands are medley-x, medley-sdl, etc. that might be best. There could also be a medley driver script to choose one of them automatically and pass the command line options to it.

[masinter]

@lassik I'm not sure how to get from where we are to the recommended layout. Do we need an 'install' script?

[nbriggs]

There are a couple of approaches, one is using the packaging tool appropriate for the OS you're making it for (e.g., https://wiki.debian.org/Packaging https://packaging.ubuntu.com/html/ https://docs.oracle.com/cd/E26502_01/html/E21383/pkgcreate.html etc.) and distributing the package blob, another is to have a "make install" target -- for most software you're building and installing you'd expect an optional "configure" step, followed by "make" (build) and "make install" steps (where you may need administrative privilege to do the install step). One can use the "configure" step to capture where the software will be installed so that if there are any path dependencies that need to be built in to the executables or installation-specific configuration files that can be done during the build step.

[nbriggs]

p.s., I don't recommend using the GNU autotools stuff, it's awkward and overkill for anything we're doing. Simple makefiles work just fine.

[lassik]

I recommend the standard:

$ ./configure
$ make
$ make install

But we don't have to use GNU autotools to implement these steps; I agree that it's overkill. We can manually implement the standard make targets and write a configure shell script.

[lassik]

I think autotools has some standard support for cross-compilation, but I'm not familiar with it.

[orcmid]

CMake seems to be the multi-platform case. Even then, likely too much.

[masinter]

These kind of configuration discussions have been with us from the beginning. I think what we need is a concrete PR to add or change the commands. The first advice we got when getting started was "simplify the build process" I thought that meant both maiko and medley.

[masinter]

please compare and contrast with @fghalasz analysis and proposal

From: Frank Halasz frank@halasz.org Sent: Thursday, December 16, 2021 2:26 PM To: Larry Masinter LMM@acm.org Cc: Bill Stumbo wstumbo@charter.net Subject: Re: medley wiki

The issue here is where to place MEDLEYDIR and MAIKODIR during install on a Linux system.

The first question to ask is whether we want these directories in a "system" or a "user" location. If in a system location, we expect the files in the directory to be read/executed but not changed or written to in normal operation (i.e., outside an explicit install operation). To write to files in a system location, you generally need privileged (root?) access. In contrast, if in a user location, we expect the user will be editing/writing files at will without any special privileges. Other differences include expectations about whether any changes are common to all users and whether the files are "system" maintained or user maintained. But in the Medley case (especially Medley inside a docker container), we are talking about essentially single user setups - so these latter differences are less important.

So the issue is whether we expect the user to be be working on (i.e., changing) the MEDLEYDIR/MAIKODIR files or just using (i.e., loading) the files and working on code/apps that run on top of the base Medley.

In setting up online.interlisp.org (aka OIO), I made an assumption that the typical OIO user would be working on MEDLEYDIR files but not on MAIKODIR files. So, I moved MEDLEYDIR into a user location and left MAIKODIR in a system location. Note that for Notecards.Online (NO), I left MEDLEYDIR and MAIKODIR in system locations and placed the Notecards code in a user location, assuming that the typical NO user would want to edit the Notecards code but not touch the Medley base code.

If we decide to go with MEDLEYDIR and/or MAIKODIR is a system location, the next question to ask is what system system location. This issue is a bit of a mess in Linux. But there is a scheme that most distributions try to follow: Filesystem Hierarchy Standard. Here is a excerpt from StackExchange that explains the FHS pretty well:

Basically (and very roughly), the standard paths on Linux are:  /bin & /sbin are for vital programs for the OS, sbin being for administrators only ;  /usr/bin & /usr/sbin are for not vital programs, sbin being for administrators only ;  /var is for living data for programs. It can be cache data, spool data, temporary data (unless it's in /tmp, which is wiped at every reboot), etc. ;  /usr/local is for locally installed programs. Typically, it hosts programs that follow the standards but were not packaged for the OS, but rather installed manually by the administrator (using for example ./configure && make && make install) as well as administrator scripts ;  /opt is for programs that are not packaged and don't follow the standards. You'd just put all the libraries there together with the program. It's often a quick & dirty solution, but it can also be used for programs that are made by yourself and for which you wish to have a specific path. You can make your own path (e.g. /opt/yourcompany) within it, and in this case you are encouraged to register it as part of the standard paths ;  /etc should not contain programs, but rather configurations. If your programs are specific to the services provided by the service, /srv can also be a good location for them. For example, I prefer to use /srv/www for websites rather than /var/www to make sure the directory will only contain data I added myself, and nothing that comes from software packages. My reading of this (and my experiences with Linux - mainly in the Debian/Ubuntu arena) says that MEDLEYDIR and MAIKODIR would go into /opt - preferably as something like /opt/interlisp/maiko and /opt/interlisp/medley, but could also be /opt/maiko and /opt/medley.

I have never seen a /app directory on a Linux system - but then I have mostly dealt with the Debian/Ubuntu crowd and it might be common with other lines in the Linux distribution tree. But in the current OIO docker image, I left MAIKODIR in /app since it has no real impact on OIO whether it is in /app or /opt.

If we decide to go with MEDLEYDIR and/or MAIKODIR in a user location (i.e., under $HOME), then for general "orderliness" of home directories and for purposes of OIO its important that they do into a subdirectory of $HOME (e.g., into $HOME/il/medley and/or $HOME/il/maiko). This is necessary for OIO so we can maintain independent volumes for MEDLEYDIR code and for "everything else" in $HOME. This way, we can update the Medley code to the latest without wiping out any changes that the user made in $HOME.

There is also the issue of LOGINDIR. This is where Medley stores (by default) lisp.virtualmem and the user INIT file. Right now this defaults to $HOME unless $LOGINDIR is set. This too should be in a subdirectory of $HOME and not in $HOME directly, mainly for purposes of maintaining modularity in $HOME. Typically, in Linux I think it would go into a hidden subdirectory of home such as $HOME/.medley or $HOME/.interlisp or $HOME/.il. It also seems to me that this should be a separate subdirectory from MEDLEYDIR (if MEDLEYDIR is in $HOME) for purposes of updating standard MEDLEYDIR independently of user files such as INITs. (Note currently in OIO I do make LOGINDIR be the same as MEDLEYDIR. That should change.)

I am not 100% sure of everything that LOGINDIR does in Medley. So it is possible that messing with LOGINDIR may adversely affect things other than lisp.vmem and INIT. In OIO is did move LOGINDIR to $HOME/il. And all worked ok. But I have not extensively tested this 'feature'.

Also note that LOGINDIR is not a very good name since login seems to be peripheral to what it really does in Medley. But it is probably not worth changing at this point.

Bottom line: my proposal for online and docker is: MAIKODIR is /opt/interlisp/maiko (not just /opt/maiko in case we ever want to move MEDLEYDIR to /opt) MEDLEYDIR is $HOME/il (which would be /home/medley/il - since the user in docker images is 'medley'). LOGINDIR is $HOME/.interlisp (since I think $HOME/.il would be too easily confused with $HOME/il).

[orcmid]

@masinter please compare and contrast with @fghalasz analysis and proposal

... My reading of this (and my experiences with Linux - mainly in the Debian/Ubuntu arena) says that MEDLEYDIR and MAIKODIR would go into /opt - preferably as something like /opt/interlisp/maiko and /opt/interlisp/medley, but could also be /opt/maiko and /opt/medley.

I have never seen a /app directory on a Linux system ... But in the current OIO docker image, I left MAIKODIR in /app since it has no real impact on OIO whether it is in /app or /opt.

If we decide to go with MEDLEYDIR and/or MAIKODIR in a user location (i.e., under $HOME), then for general "orderliness" of home directories and for purposes of OIO its important that they do into a subdirectory of $HOME (e.g., into $HOME/il/medley and/or $HOME/il/maiko). ... Bottom line: my proposal for online and docker is: MAIKODIR is /opt/interlisp/maiko (not just /opt/maiko in case we ever want to move MEDLEYDIR to /opt) MEDLEYDIR is $HOME/il (which would be /home/medley/il - since the user in docker images is 'medley'). LOGINDIR is $HOME/.interlisp (since I think $HOME/.il would be too easily confused with $HOME/il).

Understanding that the fall-line for Interlisp is *nix, there is some possibly-useful information for what works on Windows 10/11 (not WSL).

There is an app-folder installation procedure where an app is installed in a subfolder and the app is sandboxed there. The app can have resources, libraries (DLLs) and data files alongside its .exe code. There does not always seem to be ways to access the containing file system and the (command-line) environment. Native (win32) apps might be able to access more, but apparently .NET apps cannot.

There is an important hidden (but not system) folder, C:\user\orcmid\AppData. I think this is the top of a folder structure where app sandbox subfolders are stashed. I am not clear that is actually the case. ('C:\user\orcmidis the counterpart of~orcmid`.)

The TL;DR: I need to research this more. The app folder installation case is nice, but AppData is a mess.

---

The messes. The user home location is usually not cluttered with much. My C:\users\orcmid\ folder has a lot of .-prefixed subfolders, some stray things, and the standard user-default folders like Documents, Pictures, Music, etc. Those last might or might not be symlinks. They can be moved elsewhere, but that leaves those as empties and might sometimes be used by mistake from some apps.

The .-prefixed folders tend to be some kinds of configuration information, there even being one named .config from someone.

There are no shell scripts nor login setups nor anything like etc/ exposed in the file system.

AppData is a by-user user-writable location and it has fixed AppData\Local, AppData\LocalLow, and AppData\Roaming under which application-associated subfolders are introduced. Different organizations put subfolders in different ones, often more than one. It is a bit of a wild west in there. There are AppData\Local\Temp, AppData\LocalLow\Temp, and C:\Temp (shrugs).

Because it does not require administrator privileges, it is tempting to put apps down under AppData. I think Google Chrome did that originally and now everyone tries it. Visual Studio Code sits in there. There is also a privileged place of some sort where apps from the Microsoft Store can go. That seems more protected.

[nbriggs]

What we do for Linux may be different from what we do for other Unix systems -- /usr/local rather than /opt would be where FreeBSD expects us to install, and there's the question of whether we drop lde [and friends, likely renamed] into /usr/local/bin/ or /usr/local/maiko/bin/ -- or do ldex, ldesdl, and any other display programs that shouldn't be directly executed, go into /usr/local/libexec/maiko/ ? I'm not a big fan of /opt, and I think even Linux systems (or Linux system administrators...) are tending towards using /usr/local, though that usage is not yet codified in the FHS.

We used to use /usr/local/lde/site-init.lisp as the canonical site initialization file.

The default working directory for a user is kept in the lisp variable LOGINHOST/DIR and isn't available to maiko at the time it is reading or creating the lisp.virtualmem. I agree that the lisp.virtualmem and user greet files should go in a subdirectory of $HOME rather than at the top level -- nobody's home directory is owned by any single application these days. USERGREETFILES is set up to provide a list of patterns for finding them -- it could likely be different for OIO than for a desktop install. It would normally be set up by the site init file (I think by default it's NIL, per sources/APUTDQ).

MEDLEYDIR is a convenience for locating the root of the sources, lispusers, library, fonts, and any other directories that are standard parts of the release, and then setting up the variables that the rest of the system uses (DIRECTORIES FONTDIRECTORIES WHEREIS.HASH (or whatever it is called today)). I agree that the average user shouldn't be overwriting the released parts of the system, so those could/should be installed in a system location. As I've said before, I don't think one should expect to git clone medley directly into the install location! clone, [configure], make, make install is one way, building OS packages is another way (say, Debian packages, macOS installation packages for, say, /Applications/Interlisp with everything that's static kept under there). macOS has similar concepts to the Windows 10/11 stuff that Dennis mentioned -- though they work better for people who didn't grow up with *nix hierarchies.

[fghalasz]

I am comfortable with placing the Maiko bin files into /usr/local (as opposed to /opt/...). Putting lde into /usr/local/bin makes sense. ldex, etc into /usr/local/libexec also makes logical sense but not sure I have ever seen a /usr/local/libexec (for what its worth). Putting all Maiko bins in /usr/local/maiko (or subdirs thereof) would also work and make sense.

This would require renaming at least lde to something less likely to have name collisions.

This would also involve writing a "make install" for Maiko, at least for the Linux installs. The "make install" would obviate the need for the os.arch directories and at least on Linux you could simplify run-medley a bit. The Docker image builds would need to be updated to run make install.

On the Medley front, I am okay with moving online Medley to a system location. Then for developers working on the actual Interlisp code would keep their own separate source directories much like Ron now does to handle the git and branching issues. (Alternatively, I could use two different docker images - one for Medley devs with the Medley code in user space and one for everyone else with the Medley code in system space.)

I still think that /opt is the best system place for the Medley code - its app code that doesn't quite fit the Linux template - just what /opt was designed for. But if that's objectionable to too many, we could also just put it in /usr/local/medley (or /usr/local/interlisp). In any case the install script would be a simple "git clone --depth=1 ...:medley && rm -rf medley/.git".

If we go the /usr/local/medley route, then we should probably merge the install locations for medley and maiko and just put lde etc into /usr/local/medley/bin (or /usr/local/interlisp/bin).

-- Frank

[orcmid]

@nbriggs ... building OS packages is another way (say, Debian packages, macOS installation packages for, say, /Applications/Interlisp with everything that's static kept under there). macOS has similar concepts to the Windows 10/11 stuff that Dennis mentioned -- though they work better for people who didn't grow up with *nix hierarchies.

Along with release of Windows 11, there are the results of what was called Project Reunion (now neutered to "Windows App SDK") for coalescing the many ways of building GUI applications and also provide a layer of abstraction above changes at the Windows API/SDK level that make for insane dependency discovery/management. This works retroactively to some level of Windows 10. Another aspect of it is a way to have light-weight distributions, practically equivalent to XCopy (or installing by unzipping). It seems this is to make it easier to introduce apps into the Microsoft Store and also be able to install them directly, as is common for in-enterprise distributions. That is about the sandboxed apps that I referred to. It is related somehow to MSIX, a new/recent installer format.

I don't have enough understanding on how MSIX simplifies packaging of releases while getting the OS to do nice things, including handling of updates. Also, now there is wget, a package installer, built into current Windows 10/11 in an effort to make Windows more modular and not require big wholesale synchronized feature updates.

---

I should qualify my interest here. I have been fiddling around to see if I can build maiko using native windows tools. I have been working on facilitating that for some game development projects and (other) neophytes interested in the same sort of thing. I am dusting off my beginner mind to make better on-ramps and guard rails than found in some otherwise-promising on-line courses. That involves a different (daresay simpler) approach to graphics atop OpenGL inspired by something Borland did in the MS-DOS/Windows overlap period before they stumbled. I have become a fan of raylib and perhaps raygui for this. That probably does not work at getting "direct" (encapsulated) GUI presence of medley/maiko rather than by X11, and I am not pursuing this maiko tinkering with all my heart. I would claim victory if I managed to port Notecards that way :).

[orcmid]

In a buddy call today, my pal William Anderson was looking for some of his MacOS code to move to a replacement machine. He looked in the obvious user, opt, places. There were a few of them, all empty. It turned out that HomeBrew has itself down one of those folder paths and puts all of the installs it does beneath that folder. I suspect this has something to do with dependency management and updating.

I wonder if the advent of package managers has disrupted some traditional conventions.

[nbriggs]

It turned out that HomeBrew has itself down one of those folder paths and puts all of the installs it does beneath that folder.

Yeah, /usr/local/Cellar, because, y'know, the cellar is where you keep your wine casks/beer kegs... I guess. It makes links from /usr/local/bin for executables and a link for every package in /usr/local/opt (what??) back into the Cellar. Every package manager wants to pick its own distinct place to stash things so that it doesn't collide with any of the other ones or any "standard" installation places. As they say, "standards are great, there are so many to choose from".

This is one reason I refuse to use "homebrew" or "macports".

[masinter]

Our plan was to follow @fghalasz setting up the online container and push that back into the main medley repository as the 'canonical' way to organize files at least on linux. we're still a ways from the desirable configure, make, make install ideal -- but can we get there?

[masinter]

Recent commits have this well underway with installers...

what's left?

[nbriggs]

An install target for Maiko's "make" and "CMake", and probably "make" targets for Medley to do the loadups instead of/augmenting the scripts and an "install" target. If you've modified a Lisp file that you know is part of the loadup, then you should be able to do "make && make install" and get the new sysouts (lisp, full, etc.) installed into the "right" place (/usr/local/...) along with the site init file, and the run-medley script (which should probably be renamed "medley"). These should be consistent with what the pre-built installers do.

[masinter]

I think the various scrips installers solve the problems raised here. Closing.