Trying to load quickload the library exhausts heap on M2 Mac

[Kamefrede]

OS: macOS 14 running on an M2 Macbook Air

Using the prebuild branch (so as to not compile llvm from scratch) and trying to load the library with quicklisp (ql:quickload :claw-raylib) as per the instructions yields a heap exhaustion error and drops down the lisp repl to ldb.

Log below:

❯ sbcl
This is SBCL 2.3.8, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
* (ql:quickload :claw-raylib)
To load "claw-raylib":
  Load 1 ASDF system:
    claw-raylib
; Loading "claw-raylib"
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
.................Heap exhausted during garbage collection: 64 bytes available, 96 requested.
        Immobile Object Counts
 Gen layout fdefn symbol   code  Boxed   Cons    Raw   Code  SmMix  Mixed  LgRaw LgCode  LgMix Waste%       Alloc        Trig   Dirty GCs Mem-age
  2      0      0      0      0   3951    500    536      9    153     64      5      0     11    0.5   341037440    10737418    5229   0  0.8396
  3      0      0      0      0   5309    701    747     65    217     75      5      0     33    0.3   467144432     2000000     829   0  0.0000
  4      0      0      0      0      0      0      0      0      0      0      0      0      0    0.0           0     2000000       0   0  0.0000
  5      0      0      0      0      0      0      0      0      0      0      0      0      0    0.0           0     2000000       0   0  0.0000
  6      0      0      0      0    165     47     99    171     33      5      5      0     33    2.5    35650768     2000000      90   0  0.0000
  7      0      0      0      0   2528    305    409      8    102     32     20      0     41    0.5   224638704     2000000    3445   0  0.0000
Tot      0      0      0      0  11953   1553   1791    253    505    176     35      0    118    0.5  1068461072 [99.5% of 1073741824 max]
GC control variables:
   *GC-INHIBIT* = true
   *GC-PENDING* = true
   *STOP-FOR-GC-PENDING* = false
Collection trigger variables:
   dynamic_space_size = 1073741824
   bytes_allocated = 1068461072
   auto_gc_trigger = 892992467
   bytes_consed_between_gcs = 53687091
fatal error encountered in SBCL pid 5950 pthread 0x1d7a3d300:
Heap exhausted, game over.

Welcome to LDB, a low-level debugger for the Lisp runtime environment.
(GC in progress, oldspace=2, newspace=7)
ldb>

[bohonghuang]

By default, cffi-object performs object allocation optimization when generating function definitions. This requires a large amount of heap memory on SBCL. Therefore, you have the following options:

1. Disable this optimization by executing (setf cobj::*optimize-object-allocation-p* nil) before (ql:quickload :claw-raylib).
2. Increase the available heap memory by running SBCL with the --dynamic-space-size 4096 parameter when loading claw-raylib for the first time. Once claw-raylib is compiled into FASL, you can run it with the default heap memory size.

[Kamefrede]

❯ LD_LIBRARY_PATH="$LD_LIBRARY_PATH;/Users/emanuelferraz/common-lisp/claw-raylib/lib" sbcl --dynamic-space-size 4096
This is SBCL 2.3.8, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
* (ql:quickload :claw-raylib)
To load "claw-raylib":
  Load 1 ASDF system:
    claw-raylib
; Loading "claw-raylib"
..........................
debugger invoked on a TYPE-ERROR @70035F113C in thread
#<THREAD "main thread" RUNNING {7012490563}>:
  The value
    (CFFI::EXPLODE-PATH-ENVIRONMENT-VARIABLE "LD_LIBRARY_PATH")
  is not of type
    (OR STRING SYMBOL CHARACTER)
  when binding SB-IMPL::STRING2

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [TRY-RECOMPILING              ] Recompile library and try loading it again
  1: [RETRY                        ] Retry
                                     loading FASL for #<CL-SOURCE-FILE "claw-raylib" "library">.
  2: [ACCEPT                       ] Continue, treating
                                     loading FASL for #<CL-SOURCE-FILE "claw-raylib" "library">
                                     as having been successful.
  3:                                 Retry ASDF operation.
  4: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  5:                                 Retry ASDF operation.
  6:                                 Retry ASDF operation after resetting the
                                     configuration.
  7: [ABORT                        ] Give up on "claw-raylib"
  8: [REGISTER-LOCAL-PROJECTS      ] Register local projects and try again.
  9:                                 Exit debugger, returning to top level.

(SB-KERNEL:TWO-ARG-STRING= "/Users/emanuelferraz/common-lisp/claw-raylib/lib/" (CFFI::EXPLODE-PATH-ENVIRONMENT-VARIABLE "LD_LIBRARY_PATH")) [external]

After the most recent commit in the prebuild branch and with some manual tinkering (i'll submit a new issue after this comment) with compiling, it now fails on this stage, trying to read the type of the variable it returns as this:

* (type-of (CFFI::EXPLODE-PATH-ENVIRONMENT-VARIABLE "LD_LIBRARY_PATH"))
CONS

[bohonghuang]

Can you provide the complete error stack trace? This seems to be an internal issue with CFFI. library.lisp is mainly responsible for locating dynamic libraries and opening them. It compiles and runs correctly on Windows and GNU/Linux, but it haven't been tested on macOS yet.

[bohonghuang]

After examining the source code of CFFI, my suspicion that the issue lies with CFFI has increased. Can you execute the following form to override the original definition of CFFI and retry compiling claw-raylib?

(defparameter cffi::*foreign-library-directories*
  (if (featurep :darwin)
       (list (explode-path-environment-variable "LD_LIBRARY_PATH")
        (explode-path-environment-variable "DYLD_LIBRARY_PATH")
        (uiop:getcwd)
        (darwin-fallback-library-path))
      '())
  "List onto which user-defined library paths can be pushed.")

[chip2n]

@bohonghuang FYI, I ran into the same problem on my Arch Linux box, except it was a bit more nefarious because Quicklisp just froze without any error indication (even with verbose t). This was using the prebuild branch. Using --dynamic-space-size 8192 solved the issue (didn't try a lower amount). Maybe a note in the README would be good (I can make a PR if you don't have time).

[bohonghuang]

Maybe a note in the README would be good

I will add a statement in the README for SBCL. Thank you for your feedback.