Make MAGICL Lisp-only by default, introduce MAGICL/EXT-* systems

[stylewarning]

One of the biggest pain points of MAGICL is having to install LAPACK and GFORTRAN and etc. It's transitively difficult to use projects that depend on MAGICL as well, like quilc.

I propose that we split MAGICL into a pure-Lisp and non-Lisp-acceleerated double system. Specifically, I think there should be the following rough divisions:

Make the MAGICL system Lisp-only

We have a high-level framework for representing tensors, thanks to @colescott. The rudimentary linear algebra operations are also not too difficult to implement in Lisp. These would be:

• matrix-vector and matrix-matrix multiplicatons
• linear system solving
• a few factorization routines
• pointwise/broadcasting operations
• a few potpourri ops, like kron

We don't need to have mega high-performance implementations in Lisp, just enough to make the base library useful.

Critically, however, we remove all LAPACK stuff from the MAGICL system, and shovel it somewhere else as described next...

Put non-Lisp bindings into a new system called MAGICL/EXT-XXX

Specifically, I imagine MAGICL/EXT-BLAS, MAGICL/EXT-LAPACK, and MAGICL/EXT-EXPOKIT. The / is used because ASDF understands it as a "sub-system" of sorts, and EXT- stands for "extension".

These systems would contain two main attractions:

1. The raw bindings
2. Extensions to symbols in the MAGICL package

By point 2, I mean that generic functions that were responsible for some operations can have methods that are defined or even re-defined by loading this system, thus "accelerating" the pure-Lisp-based routines above with Fortran/C-written ones.

One open question here is: How is new functionality introduced? By way of a new package? Or exporting new symbols from MAGICL? One possibility is that the generic functions are defined in MAGICL, but no methods are adorned. (Maybe a default "not implemented" error or something is specialized on T.)

Create a MAGICL/FANCY meta-package

I don't actually know what to call it, but MAGICL/FANCY is a dummy system that depends on all of the MAGICL/EXT-XXX extension systems. If you want a souped up MAGICL, load MAGICL/FANCY.

Add a *PREFER-LISP* variable

This is optional, but it would be nice if one could prefer Lisp implementations of functions when possible, and *PREFER-LISP* could control that.

So now what?

I'd love comments and thoughts on the proposal. MAGICL development has been slow and steady, and having worked with it for a while, I think the above represent a good next step.

[stylewarning]

CC @kilimanjaro @colescott @jmbr @notmgsk

[braised-babbage]

Broadly speaking I think this is a good idea. Two quick remarks

1. We should expect some nontrivial changes in the future to the API exposed by the base magicl package: for example, we have had some discussion around support for block matrices, we may wish to also support sparse tensors of some form, etc. The naive thing (which is not unreasonable) is to just say that we will deal with such changes when they come, and perhaps they involve changing the same volume of code but now split over several systems. Another approach is to try to get "ahead of" future changes by more actively anticipating what they might look like or what they mean for the rough categorization of operations you presented (matrix-vector and matrix-matrix multiplicatons, linear system solving, etc...).

2. As to how new functionality gets introduced by extensions, I think having it reside in a separate package sounds quite reasonable, with the accelerated implementations of existing functionality corresponding to methods on magicl generics. I don't like the idea of having magicl extensions introducing new symbols to the magicl package, since a) this requires that the extensions negotiate between themselves as to who owns what new symbol, and b) it's would not be immediately evident from user code as to whether "vanilla" magicl or "extended" magicl are required. Of course there is still an element of (a) in having methods on magicl generics, since two extensions could in principle provide methods with the same type signatures, but the semantics of these implementations is defined by the base magicl package, so the choice of one vs another shouldn't affect the meaning or correctness of a user program.

[notmgsk]

Something like:

• magicl/base defines the interface, and everything is (error 'not-implemented)
• magicl/reference is the lisp implementation of the interface
• magicl/fancy is the blazing fast implementation of the interface

Then as a user I can use the appropriate package, without having to worry about rewriting all magicl.reference:... to magicl.fancy:... when I want to switch.

[jmbr]

FWIW, I'm in favor of completely decoupling the BLAS/LAPACK bindings from the rest of MAGICL. People who want to use just that should not feel like they have to buy into the whole MAGICL API.

I'm surprised about the pain points you report. I'd expect most users to have some set of linear algebra libraries in their system. This is certainly true for Mac OS, which bundles the Accelerate framework. Typical Linux users interested in quantum computing will likely already have NumPy and SciPy installed, which have BLAS/LAPACK as prerequisites, usually satisfied via ATLAS (comes with vanilla distributions), Intel MKL (bundled with Conda), or OpenBLAS (for the discerning users out there).

As for GNU Fortran, my recollection is that it's only necessary for the (minority of) users that want to rebuild the interfaces from scratch. By the way, why write the linear algebra routines in Lisp by hand? F2CL already bundles many of them.

To summarize, I believe the root problem might be software configuration/distribution and, in that case, it would be better to address that directly. The effort spent on reference implementations of algorithms, etc. could be channeled into making MAGICL / Common Lisp more competitive as a language for numerical computing and experimentation (e.g., supporting CUDA, interfacing to PyTorch, etc.).

Just my $2E-2.

[colescott]

Can we close this now that #125 merged?