Please include an arm64 build of libmd

[nathanvy]

I'm unable to properly load lisp-stat on Apple Silicon because this particular package fails to load, because the packaged version of libmd is compiled for x86_64.

A quick demonstration

Remove the cephes directory from quicklisp:

$ rm -rf cephes.cl-20221106-git/ 

And then:

$ sbcl
This is SBCL 2.3.4, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
* (ql:quickload :lisp-stat)
To load "lisp-stat":
  Load 1 ASDF system:
    lisp-stat
; Loading "lisp-stat"
To load "cephes":
  Load 2 ASDF systems:
    asdf cffi
  Install 1 Quicklisp release:
    cephes.cl
; Loading "cephes"
.............Library #P"/Users/nathan/quicklisp/dists/quicklisp/software/cephes.cl-20221106-git/scipy-cephes/libmd.dylib" exists, skipping build

debugger invoked on a CFFI:LOAD-FOREIGN-LIBRARY-ERROR in thread
#<THREAD "main thread" RUNNING {70087906B3}>:
  Unable to load any of the alternatives:
   ("libmd"
    #P"/Users/nathan/quicklisp/dists/quicklisp/software/cephes.cl-20221106-git/scipy-cephes/libmd.dylib")

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY                        ] Try loading the foreign library again.
  1: [USE-VALUE                    ] Use another library instead.
  2: [TRY-RECOMPILING              ] Recompile init and try loading it again
  3: [RETRY                        ] Retry
                                     loading FASL for #<CL-SOURCE-FILE "cephes" "init">.
  4: [ACCEPT                       ] Continue, treating
                                     loading FASL for #<CL-SOURCE-FILE "cephes" "init">
                                     as having been successful.
  5:                                 Retry ASDF operation.
  6: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  7:                                 Retry ASDF operation.
  8:                                 Retry ASDF operation after resetting the
                                     configuration.
  9: [ABORT                        ] Give up on "cephes"
 10:                                 Give up on "lisp-stat"
 11: [REGISTER-LOCAL-PROJECTS      ] Register local projects and try again.
 12:                                 Exit debugger, returning to top level.

(CFFI::FL-ERROR "Unable to load any of the alternatives:~%   ~S" ("libmd" #P"/Users/nathan/quicklisp/dists/quicklisp/software/cephes.cl-20221106-git/scipy-cephes/libmd.dylib"))
   source: (ERROR 'LOAD-FOREIGN-LIBRARY-ERROR :FORMAT-CONTROL CONTROL
                  :FORMAT-ARGUMENTS ARGUMENTS)

Furthermore:

$ file libmd.dylib 
libmd.dylib: Mach-O 64-bit dynamically linked shared library x86_64
$ uname -a
Darwin polaris.local 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:23 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6020 arm64

Note here in particular the mismatch between the dylib arch and that reported by uname.

Workaround

If I manually replace the included dylib with an arm64 build from homebrew, everything builds correctly.

Discussion

I'm not sure how you want to approach this because you can't very well include two files with identical names. I'm not particularly familiar with ASDF and whether or not it allows us to include a file conditional on the target OS, but I'm able to conduct some tests on my end if you don't have access to an arm64 machine.

[snunez1]

I'd love it if you could experiment, as I don't have access to an arm64 machine. I suspect that it's possible to conditionalise somehow, and it would be a useful pattern to have in other cases where we distribute binary shared libraries, like BLAS.

My *features* has :x86-64 in it; I suspect that there's a similar symbol in an arm64 implementation that could be used to detect the right architecture/library to load.

[snunez1]

You know, it occurred to me we might be overthinking this. There's no reason the libraries can't have different names, e.g. libmd-macos-x86-64.dylib and libmd-macos-arm64.dylib and then have the code conditionalised on the architecture in *features*. Untested stab at a solution (init.lisp):

(:darwin (:or "libmd"
              #+x86-64 #.(merge-pathnames "scipy-cephes/libmd-macos-x86-64.dylib" *compile-file-pathname*)
              #+arm-64 #.(merge-pathnames "scipy-cephes/libmd-macos-arm-64.dylib" *compile-file-pathname*)))

[nathanvy]

For completeness' sake, here is my *features* on an M2 Macbook pro, sbcl being installed from homebrew:

* *features*
(:QUICKLISP :ASDF3.3 :ASDF3.2 :ASDF3.1 :ASDF3 :ASDF2 :ASDF :OS-MACOSX :OS-UNIX
 :NON-BASE-CHARS-EXIST-P :ASDF-UNICODE :ARM64 :GENCGC :64-BIT :ANSI-CL :BSD
 :COMMON-LISP :DARWIN :IEEE-FLOATING-POINT :LITTLE-ENDIAN :MACH-O
 :PACKAGE-LOCAL-NICKNAMES :SB-CORE-COMPRESSION :SB-LDB :SB-PACKAGE-LOCKS
 :SB-THREAD :SB-UNICODE :SBCL :UNIX)

So you could branch on the contents of *features* but the reader macro seems simpler to me. The only caveat is that if libmd.dylib is not present then the makefile builds it, so editing init.lisp in such a way also requires edits to the .asd file as well as the makefile.

Fortunately, when building the library from source it compiles correctly on my arm64 machine.

So maybe the simplest solution is to not include a prebuilt .dylib at all?

[snunez1]

The reason we include the shared library is that most mac users don't have a compiler tool chain installed, and we want it to be 'ready to run' for as many as possible.

If the user does have a compiler and wants to rebuild the shared library, then I hope they have enough knowledge to edit the init.lisp and rename the file. Unless your suggestion is to also modify the makefile to detect the architecture and write out a dylib with the correct name? Actually, thinking about it, that's probably the 'correct' answer here.

[nathanvy]

Unless your suggestion is to also modify the makefile to detect the architecture and write out a dylib with the correct name?

This is what I was getting at, yes. I can send you a pull request shortly.