WIP: Quicklisp runner in cargo-make

This issue is work in progress issue expected to be submitted in https://github.com/sagiegurari/cargo-make as a followup on https://github.com/sagiegurari/cargo-make/issues/461

DISCLAIMER: I am not a lisp backend developer (i just use the language on preferred implementation atm) so the information is provided to the best of my ability and may be inaccurate, peer-reviews were made and addressed.Any relevant information/criticism is welcomed.

github_quicklisp

This is a feature request to implement quicklisp https://www.quicklisp.org/beta/ which is a deployed through loadable lisp library expected to allow for implementation-independent code in cargo-make https://github.com/sagiegurari/cargo-make which is arguably a better alternative alternative to make command reading Makefile.

Expectation

The ability to use lisp and/or common-lisp (programming language) called from cargo-make on all supported devices by rustlang and/or *lisp (https://doc.rust-lang.org/nightly/rustc/platform-support.html) with readable and fault tolerant implementation-independent implementation while not preventing implementation of other touring complete systems.

My usecase

I want to use (C)lisp on my repositories that are aiming to be cross-platform compatible with ideology to support as many devices as possible where it's not limiting me in terms of technology for things where it's not practical to implement them in rustlang (scripts).

I prefer rustlang since it allow me to outsource my code in libraries (crates) allowing for passive maintenance of my codebase which is much less efficient on C + i am fed up with keep hotfixing of C standard issues and wasting time looking for memory leaks where it seems that rustlang is not less efficient then C assuming optimization made i.e. comparing fibonacci in rustlang https://github.com/Kreyren/rustlang-fibonacci/tree/kreyren/case-study-performance-2 to C and Lisp.

Rustlang currently works on less devices compared to lisp which i want to in worst case scenario implement though lisp wrapper to read the Makefile.toml.

Issue

Currently cargo-make version =0.32.6 requires the following entry to run common lisp through Embedded Common Lisp (ecl) to use implementation independent code style:

[env]
MESSAGE = "something"

# NOTICE(Krey): You will need quicklisp installed

[tasks.kreyren]
script_runner = "ecl"
script_runner_args = [ "--norc", "--quiet", "--shell" ]
script_extension = "cl"
script = [
'''
(setf *load-verbose* nil)
(load "/home/kreyren/quicklisp/setup.lisp" :verbose nil)
(ql:quickload :uiop :silent t)

(write-line (uiop:getenv "MESSAGE"))
'''
]

to return something which is hard to read and maintain as it requires duplicate code (namely script_runner_args and lisp lines above write-line)

Where the expected is:

[env]
MESSAGE = "something"

[tasks.kreyren]
script_runner = "@quicklisp"
script = [
'''
(write-line (uiop:getenv "MESSAGE"))
'''
]

Implementation compatibility

Common lisp has many implementations alike:

CLISP (clisp) - Implementation of Common lisp written in C
CCL (Clozure CL)
Clasp - Effort to write C++/LLVM common lisp implementation
ABCL (Armed Bear CL)
ACL (Allegro CL)
LW (LispWorks)
CMUCL (Carnegie Mellon University)
MKCL (ManKai)
Scieneer CL
SICL - Work in progress implementation
Embedded Common Lisp (ECL)
Steel Bank Common Lisp (SBCL)
possibly more..

Where following are dialects of lisp that are not supported by quicklisp as quicklisp is depending on ASDF that is using CLOS that is close to impossible to port to these (See statement below):

Emacs Lisp (elisp)
Scheme
Racket - https://docs.racket-lang.org/
etc..

Where hard-coded logic is mentioned in specification: http://www.lispworks.com/documentation/HyperSpec/Body/03_ababa.htm which are implemented in quicklisp that allows to write implementation-independent common lisp (meaning that the common lisp code written will work on all other implementations).

Example of implemendation dependent code printing value of environment variable MESSAGE:

(write-line (ext:getenv "MESSAGE"))

as ext is specific to ecl and sbcl (possibly others..).

Whereas this implementation works on all implementation provided:

(load #p"~/quicklisp/setup.lisp")
(write-line (uiop:getenv "MESSAGE"))

Thus the script_runner = "@quicklisp" should be looking for executables capable of processing the runtime instead of depending only on the hard-coded.

Silencing unwanted output

By default quicklisp is outputting lots of unwanted informations:

kreyren@leonid:~$ export MESSAGE=kreyren
kreyren@leonid:~$ ecl
ECL (Embeddable Common-Lisp) 20.4.24 (git:UNKNOWN)
Copyright (C) 1984 Taiichi Yuasa and Masami Hagiya
Copyright (C) 1993 Giuseppe Attardi
Copyright (C) 2013 Juan J. Garcia-Ripoll
Copyright (C) 2018 Daniel Kochmanski
Copyright (C) 2020 Daniel Kochmanski and Marius Gerbershagen
ECL is free software, and you are welcome to redistribute it
under certain conditions; see file 'Copyright' for details.
Type :h for Help.  
Top level in: #<process TOP-LEVEL 0x7fb1d054af80>.
> (load "/home/kreyren/quicklisp/setup.lisp")

;;; Loading "/home/kreyren/quicklisp/setup.lisp"
;;; Loading #P"/usr/lib/x86_64-linux-gnu/ecl-20.4.24/asdf.fas"
#P"/home/kreyren/quicklisp/setup.lisp"
> (ql:quickload :uiop)
To load "uiop":
  Load 1 ASDF system:
    uiop
; Loading "uiop"

(:UIOP)
> (write-line (uiop:getenv "MESSAGE"))
kreyren
"kreyren"
>

To silence these on ecl it's expected to use:

--shell to silence the header including copyright
(setf *load-verbose* nil) to silence the ;; Loading .. messages from ecl
(load "/home/kreyren/quicklisp/setup.lisp" :verbose nil) to silence the `;;; Loading #P"/usr/lib/x86_64-linux-gnu/ecl-20.4.24/asdf.fas"
P"/home/kreyren/quicklisp/setup.lisp"` from quicklisp itself
(ql:quickload :uiop :silent t) to silence the loading of quicklisp in the said implementation

Additionally we need argument --norc to avoid sourcing of ~/.eclrc which could interfiere with the logic in cargo-make's script.

on clisp this is getting:

kreyren@leonid:~$ MESSAGE=kreyren clisp test.lisp 
WARNING: DEFGENERIC: redefining function DIST in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function SYSTEM-INDEX-URL in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function RELEASE-INDEX-URL in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function AVAILABLE-VERSIONS-URL in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function RELEASE in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function NAME in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function BASE-DIRECTORY in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function METADATA-NAME in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function PREFERENCE-PARENT in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function SHORT-DESCRIPTION in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function PROVIDED-RELEASES in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function PROVIDED-SYSTEMS in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function ARCHIVE-URL in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
kreyren

To silence these we need --quiet

Handling of arguments per implementation

Recommend implementing script_runner_args at the background depending on found executable namely this should be doing if the end-user did not overwrite script_runner_args already:

# ECL
script_runner_args = [ "--norc", "--quiet", "--shell" ]
script_extension = "cl"
# clisp
script_runner_args = [ "--norc", "--quiet" ]
script_extension = "cl"
# elisp - Doesn't work atm
script_runner_args = [ "--quick" ,"--script" ]
script_extension = "cl"
# sbcl
script_runner_args = [ "--no-userinit", "--script" ]
script_extension = "cl"

rlisp

rlisp was concluded to be not usable https://github.com/Kreyren/rust-lisp/actions/runs/295136199

Created https://github.com/swgillespie/rust-lisp/pull/7 to track the code usability

Filed https://github.com/swgillespie/rust-lisp/issues/6 to get more info

elisp

elisp is able to process the file using script_runner_args = [ "--quick" ,"--script" ], but does not work:

Loading /home/kreyren/quicklisp/setup.lisp... Symbol’s function definition is void: defpackage

Filed https://github.com/quicklisp/quicklisp-bootstrap/issues/21 for the elisp compatibility of quicklisp

EDIT: Is not supported

Prepending lisp code to the created scripts

For the implementation to be able to implement quicklisp we need to source quicklisp library which can be done by prepending

(load #p"~/quicklisp/setup.lisp")

assuming that it has been installed on the system.

Deployment of quicklisp

To be able to run quicklisp we need to run https://beta.quicklisp.org/quicklisp.lisp to get the backend to be used in the implementation.

FIXME: How to implement this in cargo-make?

Quicklisp goal

Allegedly the goal is to make it easy to distribute and update Lisp code over the Internet, which may interfere in presented usecase.

Caches

Worth mentioning that common-lisp is caching it's functions in ~/.cache/common-lisp which might influence the runtime as the changes might not be present in the real-time

Quicklisp compatibility with non-common lisp

This is a quote of Zach Beane (@xach) from irc.freenode.net/#lisp (was allowed to quote):

<Xach> quicklisp is a common lisp program. other lisps are not common lisp and are not compatible.
<Xach> other lisps could be made compatible. it's a lot of work. nobody has done it. common lisp has a lot of features and quicklisp uses a lot of 
them.
<Xach> also, quicklisp uses extra-standard functionality that would also need implementation - networking and filesystem work mostly.
<kreyren> Is quicklisp implemented by design to allow possible implementation of non-CL interpretations or would that require lots of rewritting?
<Xach> kreyren: it is implemented by design to take full advantage of Common Lisp and I think trying to make it work elsewhere would be difficult and not very rewarding - since it is meant to allow you to run other common lisp programs, which also are not portable to other lisps.
<Xach> "Lisp" isn't generic - there are only specifics
<Xach> I think the ideas of quicklisp are pretty portable, even if the code itself is not especially

Clisp maintenance

Based on activity of https://gitlab.com/gnu-clisp/clisp it was advised to not rely on clisp as it seems somewhat unmaintained which might be subjective assuming that there doesn't seem to be any actionable issues.

EDIT: Merging the https://gitlab.com/gnu-clisp/clisp/-/merge_requests/3 seems actionable enough assuming unmaintained.

EDIT2: requested an official statement from GNU about maintenance.

References

https://courses.cs.washington.edu/courses/cse341/04wi/lectures/14-scheme-quote.html
emacs scripting https://www.emacswiki.org/emacs/EmacsScripts
Quicklisp on github https://github.com/quicklisp
http://clhs.lisp.se/
https://web.archive.org/web/20200426054415/http://home.pipeline.com/~hbaker1/TInference.html
Rust your own lisp https://dev.to/deciduously/rust-your-own-lisp-50an
Risp https://stopa.io/post/222
rust_lisp https://crates.io/crates/rust_lisp
Ketos https://crates.io/crates/ketos

Kreyren / kreyren