Improve or remove install script and document

[akce]

These are just some thoughts on improving the install script. It's been useful for some, but has proven to be a bit brittle.

Its current behaviour is to assume only top level library files have the .sls extension. Those are run through a preprocessor, installed to the destination directory and listed in the generated compile-all script, which is then executed (in order to fully compile the libs).

The preprocessor acts like a C-preprocessor #include for the local macro include/resolve so the contents of the referenced file are inlined in the installed version of the file.

The problem this solved (as include/resolve would only ever evaluate the code at runtime) was that inlining enabled full pre-compiles, pre-compiled srfi libs started up noticably faster on slow machines and some extra errors were caught.

I think one part of this task is to try and replace all that with an include/resolve that resolves the provided path specifier at compile time and passes that to include as include can compile the sourced file.

eg, (include/resolve (("srfi" "%3a115") "regexp-impl.scm")) would resolve to:

• (include "srfi/:115/regexp-impl.scm") or
• (include "srfi/%3a115/regexp-impl.scm") depending on what's locally installed.

If that works, then there's no need to preprocess files and using these libs in-place would also have the benefits of pre-compiles.

So then the question is would an install process be needed? Or does the install process become link-dirs and maybe a compile step as well? Is it useful to install compiled binaries only?

I think the compile component of the install script would still be useful. Adding whole program optimisation files would be great for those using srfi's in compiled programs.

But the issue there is determining what are the top level library files, as those files are the inputs to compile-library.

Is it okay to restrict the .sls extension to the top level libfiles only? Is there a better way to auto-discover top level files?

Or should the compile process be driven from a file? Maybe registry-names.sls?

It would need to handle srfi's that define multiple top level libs. eg, (srfi \:146) and (srfi \:146 hash) are distinct top level interfaces provided by the one srfi.

[weinholt]

I just want to check if you're aware of the package manager Akku: https://akkuscm.org/. Chez-srfi is available from there and Akku already has solutions to the problem of library installation. (Apologies if it's not appropriate for your use case).

[akce]

Hi @weinholt, thanks for pointing out Akku.

Does Akku support pre-compiling the entire source for the srfi libs? ie, is include/resolve handled in some way such that the referenced file is compiled to file?

Full pre-compilation was the main reason that install was written and if Akku already does it, then there's probably a good case for removing the local install script.

[akce]

I have to humbly beg for forgiveness for inflicting install on everyone.

I was wrong about include/resolve.

I've just tried importing after compiling and removing all source files and there was no missing file issue. Last year, when i thought i there was a problem i remember deleting a .scm to trigger the error, but that test must've been flawed because there's no problem for me now.

I think this issue turns into a remove install, or at least clear out all the pre-processing bits and/or adapt it to allow binary only installs.

Install created a compile script, and compiling is something i'd like to keep, so maybe that can become static and any new srfi libs just get added to it.

Sorry again.

[weinholt]

Does Akku support pre-compiling the entire source for the srfi libs? ie, is include/resolve handled in some way such that the referenced file is compiled to file?

Akku does not currently have any functionality for pre-compilation, although it would be a very appreciated contribution, even if it's initially Chez-specific.

[akce]

I've begun a refactor of install in this branch.

It's now split into 3 modes:

• copy and translate source to a destination dir
• link-dirs (exactly the same as the existing link-dirs script)
• compile source at directory

The copy/translate mode is greatly simplified as it no longer preprocesses source files (doh!). Filenames at the destination are translated so include/resolve is modified to handle both % and non-% encoded filenames.

The copy mode discovers SRFI libs by looking for files/dirs starting with a % char and recursively copies those. It also copies the private directory tree and the tests directory. So anything can be inside the srfi lib dirs, install will blindly copy all the contents.

tests are copied because the current version of install copies it, but there's a case to be made for leaving it out or making it an optional copy as test_all.sh takes an srfi directory arg so it's not tied to its location.

Compile source now generates its list of compile library imports from registry-names.

This means that install no longer imposes any kind of restriction on file extensions etc, but it will need the registry file to be kept up to date. The only downside i can see so far is srfi's not in the registry won't be compiled.

I reckon these changes make install a bit more resilient. I'll keep playing around with this branch and then sort out the documentation and tests side of things if all goes well.

Do these algorithms seem okay @arcfide?

BTW, there's a few interesting things i've come across while working on this.

The registry-names are slightly out of date. Just from a quick look, \:15, \:111, and \:113 are missing.

The srfi library-name for \:115 is regex, however here and in the srfi sample implementation it's regexp.

There's a couple of SRFIs that have no named implementation (\:175 and \:219) so it's not possible to import these via a string neumonic. eg, (import (srfi \:175 ascii)).

Conversely, \:15 fluid-let only has a named library defined. ie, it's not possible to (import (srfi :15)).

I don't know if these are problems though. install could be modified to flag any of these as errors?

And just for fun, there's one accidental empty file committed: vectors.sls3a132.sls

It's an interesting idea, but general package compiling in Akku is probably a bit outside of the scope of this issue @weinholt. Maybe Akku could be modified to take advantage of whatever compile process (if any) comes from here?

[weinholt]

It's an interesting idea, but general package compiling in Akku is probably a bit outside of the scope of this issue @weinholt. Maybe Akku could be modified to take advantage of whatever compile process (if any) comes from here?

I am watching with interest. Until now I've just used scheme --compile-imported-libraries.

[arcfide]

I am likewise watching with interest. :-) My traditional use case has always been to install them into a directory where they will be searched and then let Chez compile things. I haven't found it particularly desirable to pre-compile things into an installation directory, but if this is something desirable, then I don't see why it shouldn't remain possible with the script.

As for the inconsistencies you mention, I think rather than trying to work around them with install it would be better to simply correct the errors and inconsistencies. Some of these have already been identified and fixed, such as missing names in a few places, but I'm happy to merge in more fixes to these inconsistencies.

[akce]

I'm not sure that there's anything interesting to see! :)

The compile feature is only a thin wrapper on top of (compile-imported-libraries #t) so not different to how you guys already use it except that the import list is bigger.

I'm coming around to the idea that the install script should be removed as pre-compilation is not really needed. It's thinking that comes from other language implementations like CPython.

ie, even if libs are installed to a read-only system dir like /usr/lib/csv9.5.4, Chez scheme allows for configuring (library-directories) so that the object (compile) dir can be somewhere that the user can write.

The other reason i thought of pre-compiling is to catch errors, but that's of limited use too. The pre-processed source form seems a bit stricter: these warnings aren't seen in the straight include/resolve source.

However, pre-processing should be avoided when not necessary (and it's not!) and regular use should be of the libs as they are written. If we do want pre-processed compiles as a verification step, then we could move them into the tests dir and add it to the continuous integration script. But this is a poor substitute for real tests so i'm not in favour of this.

So i reckon install should be removed and the README gets a quick writeup on installing (either in-place with a checkout and link-dirs or via Akku) and a rough guide on adding a new SRFI which would probably just be a note to remember to update the private registry files.

[arcfide]

That doesn't sound horrible to me. I'm always in favor of finding ways to simplify the whole structure.