BLUF: Maybe we shouldn't parse Fortran and generate bindings anymore, and delete that code completely.
Details:
MAGICL was founded on the idea that we could generate easy-to-use bindings to BLAS and LAPACK so that we could take advantage of those libraries with little work. However, it has proven to be a lot of work, for reasons I enumerate below.
1. We don't actually use a majority of the existing bindings
Perhaps the most damning is that we don't actually use the bindings we make. We have hundreds of bindings, but only a handful are actually put to good use.
1a. Many bindings we don't use because we don't support the data format
As a sort of add-on to the last point, we don't support the variety of data formats supplied by BLAS natively as MAGICL structures. This makes their immediate use... useless.
2. Compiling the bindings is notoriously slow
Everyone who works with MAGICL knows the BLAS/LAPACK compilation takes forever. We even needed to make special provisions for difference compilers (ECL, SBCL) for these insanely large compilation units.
3. The Fortran sources change and break the parser
I think the Fortran sources to the reference BLAS/LAPACK implementations have changed at least twice, breaking our homebrew Fortran parser. We've had to implement workarounds in these cases.
4. Bindings are difficult to call
The bindings themselves are often quite difficult to call. We have two bindings per BLAS function, a super low-level binding (let's call it a Level-0 binding) that's just passing pointers, and a slightly higher level one (let's call it a Level-1 binding) that sorta-kinda mimics CFFI. Remember, Fortran is pass-by-reference, so we do indeed need to be passing pointers around.
However, in practice, the data structures you send in aren't always easy. For instance, it's sometimes the case you need to send in a collection of overlapping pointers. What we end up doing in practice is re-creating the Level-1 binding on a by-need basis in the codebase that's using MAGICL. We did this for the ZUNCSD function, for instance.
Perhaps it was too lofty a goal to imagine we would generate easy-to-use bindings.
Conclusion / call to action?
I think we should discuss the following points of action:
To eliminate the Fortran parser completely.
To eliminate all of the existing bindings.
To write bespoke bindings by hand for each of the functions we do use. One binding per file.
To eliminate the "LAPACK search" functions.
To document the bespoke bindings' existence in popular BLAS implementations.
I want Lisp to have BLAS/LAPACK "for free", but it just seems like our approach is not the way to accomplish that.
BLUF: Maybe we shouldn't parse Fortran and generate bindings anymore, and delete that code completely.
Details:
MAGICL was founded on the idea that we could generate easy-to-use bindings to BLAS and LAPACK so that we could take advantage of those libraries with little work. However, it has proven to be a lot of work, for reasons I enumerate below.
1. We don't actually use a majority of the existing bindings
Perhaps the most damning is that we don't actually use the bindings we make. We have hundreds of bindings, but only a handful are actually put to good use.
1a. Many bindings we don't use because we don't support the data format
As a sort of add-on to the last point, we don't support the variety of data formats supplied by BLAS natively as MAGICL structures. This makes their immediate use... useless.
2. Compiling the bindings is notoriously slow
Everyone who works with MAGICL knows the BLAS/LAPACK compilation takes forever. We even needed to make special provisions for difference compilers (ECL, SBCL) for these insanely large compilation units.
3. The Fortran sources change and break the parser
I think the Fortran sources to the reference BLAS/LAPACK implementations have changed at least twice, breaking our homebrew Fortran parser. We've had to implement workarounds in these cases.
4. Bindings are difficult to call
The bindings themselves are often quite difficult to call. We have two bindings per BLAS function, a super low-level binding (let's call it a Level-0 binding) that's just passing pointers, and a slightly higher level one (let's call it a Level-1 binding) that sorta-kinda mimics CFFI. Remember, Fortran is pass-by-reference, so we do indeed need to be passing pointers around.
However, in practice, the data structures you send in aren't always easy. For instance, it's sometimes the case you need to send in a collection of overlapping pointers. What we end up doing in practice is re-creating the Level-1 binding on a by-need basis in the codebase that's using MAGICL. We did this for the
ZUNCSDfunction, for instance.Perhaps it was too lofty a goal to imagine we would generate easy-to-use bindings.
Conclusion / call to action?
I think we should discuss the following points of action:
I want Lisp to have BLAS/LAPACK "for free", but it just seems like our approach is not the way to accomplish that.