search

digikar99 / py4cl2-cffi

CFFI based alternative to py4cl2
39 stars 9 forks source link

Porting to Allegro CL 11 / MacOS 14 / Apple silicon (M2) #13

Open LEHunter opened 4 months ago

LEHunter commented 4 months ago

As a happy user of py4cl2 who wants to move some large arrays around, I thought I might try porting py4cl2-cffi to Allegro CL 11 / MacOS 14 / Apple silicon (M2). I've made some progress, but also hit a dead end. I'm trying to install via quicklisp.

First issue was this error:

While file-compiling #'"An Anonymous Function" in

P"/Users/hunter/quicklisp/dists/quicklisp/software/py4cl2-cffi-20231021-git/src/numpy-installed-p.lisp"

; between file character positions 0 (inclusively) and 3 (exclusively): Error: Package "CL" not found.

The file just contains "CL:T". ACL doesn't define CL, and T is defined in every package. I'm using a case sensitive lisp, so I just replaced the contents of the file with "t". Works, although I'm not sure how that file gets created, so can't completely fix things.

Next up was

Subprocess with command "gcc -I/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -I/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -I'/Users/hunter/quicklisp/dists/quicklisp/software/py4cl2-cffi-20231021-git/src/' -c -Wall -Werror -fpic py4cl-utils.c && gcc -shared -o libpy4cl-utils.so py4cl-utils.o" exited with error code 1

Not sure what configuration changes would be necessary to run this in the correct directory, but compiling manually after substituting the full pathname "~/quicklisp/dists/quicklisp/software/py4cl2-cffi-20231021-git/src/py4cl-utils.c" worked.

Next problem was that modern MacOS doesn't provide endian.h So, following https://stackoverflow.com/questions/20813028/endian-h-not-found-on-mac-osx/52904079#52904079 I put https://gist.github.com/dendisuhubdy/19482135d26da86cdcf442b3724e0728 into /usr/local/include/endian.h and that worked.

Next problem was that the includes of numpy/ndarraytypes.h and numpy/arrayobject.h in py4cl-utils.c failed. I used (py4cl2:raw-pyeval "np.get_include()") to discover that the files were in ~/mambaforge/lib/python3.10/site-packages/numpy/core/include" so I could fix that manually, but didn't actually fix the configuration issue.

Finally, the linker complained about a bunch of missing symbols for the arm64 architecture, and here I am out of my depth. Error is below, any suggestions much appreciated.

gcc -I/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -I/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -I'/Users/hunter/quicklisp/dists/quicklisp/software/py4cl2-cffi-20231021-git/src/' -c -Wall -Werror -fpic py4cl-utils.c && gcc -shared -o libpy4cl-utils.so py4cl-utils.o Undefined symbols for architecture arm64: "_PyCapsule_GetPointer", referenced from: import_array in py4cl-utils.o "_PyCapsule_Type", referenced from: __import_array in py4cl-utils.o "_PyErr_Format", referenced from: import_array in py4cl-utils.o import_array in py4cl-utils.o __import_array in py4cl-utils.o import_array in py4cl-utils.o "_PyErr_Print", referenced from: _import_numpy in py4cl-utils.o "_PyErr_SetString", referenced from: _import_numpy in py4cl-utils.o import_array in py4cl-utils.o __import_array in py4cl-utils.o import_array in py4cl-utils.o "_PyExc_AttributeError", referenced from: import_array in py4cl-utils.o "_PyExc_ImportError", referenced from: _import_numpy in py4cl-utils.o "_PyExc_RuntimeError", referenced from: __import_array in py4cl-utils.o "_PyImport_ImportModule", referenced from: import_array in py4cl-utils.o "_PyObject_GetAttrString", referenced from: import_array in py4cl-utils.o "__Py_Dealloc", referenced from: import_array in py4cl-utils.o __import_array in py4cl-utils.o __import_array in py4cl-utils.o ld: symbol(s) not found for architecture arm64 clang: error: linker command failed with exit code 1 (use -v to see invocation) src %

LEHunter commented 4 months ago

Quick followup. The problems with numpy/ndarraytypes.h and numpy/arrayobject.h were because of a configuration error on my part (conda and brew not playing well together). Everything else remains.

digikar99 commented 4 months ago

As a happy user of py4cl2 who wants to move some large arrays around.

py4cl2 pickles the arrays and then transfers py4cl2 sends and receives the larger arrays as npy files, so it could be sufficient. But I understand, it can still be too slow.

I'm using a case sensitive lisp.

Ah, I'd love to support a case sensitive readtable! I will check and fix the issues case-sensitivity creates in about 1-2 days.

I'm not sure how that file gets created, so can't completely fix things.

The file src/numpy-installed-p is created by src/shared-objects.lisp. It uses it as a auto-setup permanent configuration file to check and store whether numpy is installed. Depending on whether numpy is installed, the py4cl-utils.so (or equivalent) might need to be recompiled. The path to the numpy header files itself changes. This actually needs to be improved to handle different numpy versions in different environments.

For the linker error, I'm hoping to find a cloud service to test py4cl2-cffi on Apple systems. As a guess, you can try dumping the py4cl-utils.o using objdump or equivalent to see if the missing symbols it complains about are actually missing. If they are, that makes me curious why things are working on linux. Perhaps, on linux, there are no complaints because the other libraries are already loaded on SBCL through (load-python-and-libraries). If so, we might need to supply the information obtained through the python-config program in the py4cl2-cffi/config package to the step involved in the compilation of libpy4cl-utils.so itself. This would be the compile-utils-shared-object in shared-objects.lisp. Also, shouldn't that be libpy4cl-utils.dylib? Or do both libpy4cl-utils.so and libpy4cl-utils.dylib mean the same on mac?

LEHunter commented 4 months ago

Thanks for the quick response!

I'll take a look at the linker issue tomorrow, but for now, I think you should change

(format nil "CL:~A" numpy-installed-p)

in src/shared-objects.lisp

to be (write-to-string numpy-installed-p :readably t)

the CL:~A is not portable.

digikar99 commented 4 months ago

Hi! The issue about numpy-installed-p should be fixed now.

digikar99 commented 4 months ago

I have made some progress in the macos branch - the CI uses macos-14, which according to github is M1. Let me know if you can get it working to a slightly better extent. It still needs some fixes though.

digikar99 commented 4 months ago

I got hold of a MacOS at 5 USD for a week from hostmyapple. I can confirm that all-but-one tests pass on this, if you pass --dynamic-space-size 2560 or better option to SBCL.

User-6914:~ User$ sw_vers
ProductName:        macOS
ProductVersion:     14.4
BuildVersion:       23E214
User-6914:~ User$ uname -a
Darwin User-6914 23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:31 PST 2024; root:xnu-10063.101.15~2/RELEASE_X86_64 x86_64

For reasons I'm unable to debug, on github actions:

LEHunter commented 4 months ago

How much would it cost? I’d probably be willing to buy… I’m running macOS-14 on an M2.

On Apr 24, 2024, at 7:01 PM, Shubhamkar Ayare @.***> wrote:

I got hold of a MacOS at 5 USD for a week from hostmyapple https://www.hostmyapple.com/. I can confirm that all-but-one tests pass on this, if you pass --dynamic-space-size 2560 or better option to SBCL.

User-6914:~ User$ sw_vers ProductName: macOS ProductVersion: 14.4 BuildVersion: 23E214 User-6914:~ User$ uname -a Darwin User-6914 23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:31 PST 2024; root:xnu-10063.101.15~2/RELEASE_X86_64 x86_64 For reasons I'm unable to debug, on github actions:

macos-12 (x86_64) freezes while compiling the tests. macos-14 (arm64) crashes. I wonder if I'm making some assumptions about pointers that are true on x86_64 but not on arm64. I might also check if I can get py4cl2-cffi to work on an arm64 linux device. Beyond that, an M1 mac in the cloud looks a bit too expensive. — Reply to this email directly, view it on GitHub https://github.com/digikar99/py4cl2-cffi/issues/13#issuecomment-2076359924, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACWZKN7P67VBLF5XKJKYSTY7CE2NAVCNFSM6AAAAABGRYXONGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZWGM2TSOJSGQ. You are receiving this because you authored the thread.

digikar99 commented 4 months ago

The cheapest I see on HostMyApple is 69.99 USD/month. At those price points, I might as well buy myself an M1 or M2 in the upcoming year once I benchmark against my current i7-8750H to see if it's worth an upgrade :).

I'm certain there are other options out there, but I haven't checked which turns out to be the cheapest. There are other projects I'd like to support on MacOS and not just py4cl2-cffi. Setting up a sponsor might be the best going forward.

LEHunter commented 4 months ago

I’d pay $70 for the port. And I might also have other projects.

On Apr 24, 2024, at 7:16 PM, Shubhamkar Ayare @.***> wrote:

The cheapest I see on HostMyApple https://www.hostmyapple.com/dedicated-mac-hosting is 69.99 USD/month. At those price points, I might as well buy myself an M1 or M2 in the upcoming year once I benchmark against my current i7-8750H to see if it's worth an upgrade :).

I'm certain there are other options out there, but I haven't checked which turns out to be the cheapest. There are other projects I'd like to support on MacOS and not just py4cl2-cffi. Setting up a sponsor might be the best going forward.

— Reply to this email directly, view it on GitHub https://github.com/digikar99/py4cl2-cffi/issues/13#issuecomment-2076375502, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACWZKJB37JZ7JYULAOYQL3Y7CGUNAVCNFSM6AAAAABGRYXONGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZWGM3TKNJQGI. You are receiving this because you authored the thread.

digikar99 commented 4 months ago

Alright, I'll set up a sponsor over the upcoming week or two. And will also check if py4cl2-cffi works on linux arm64 before that.

LEHunter commented 4 months ago

🤙. (Which is Hawaiian for 👍).

On Apr 24, 2024, at 7:30 PM, Shubhamkar Ayare @.***> wrote:

Alright, I'll set up a sponsor over the upcoming week or two. And will also check if py4cl2-cffi works on linux arm64 before that.

— Reply to this email directly, view it on GitHub https://github.com/digikar99/py4cl2-cffi/issues/13#issuecomment-2076399706, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACWZKJ5VZ7GV6RXNDQALODY7CIG5AVCNFSM6AAAAABGRYXONGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZWGM4TSNZQGY. You are receiving this because you authored the thread.

digikar99 commented 4 months ago

I just tested this on aarch64 via AnLinux (proot) on Termux on Android:

root@localhost:~# uname -a
Linux localhost 4.14.190-26178195-abT865XXS6DWH1 #2 SMP PREEMPT Thu Aug 3 10:33:12 KST 2023 aarch64 aarch64 aarch64 GNU/Linux

The good news is that all the tests from py4cl2-cffi-tests pass out of the box without any crashes. But that raises the concern that the crashing behavior on M1 MacOS 14 on Github Actions might be harder to debug than I initially thought.

The tests themselves pass on amd64 MacOS 14 as I tried on a HostMyMac instance the other day. Consistent freezing on github actions is reported to be not very uncommon.

M1 MacOS and their successors are the specific concerns then. Could you check on both ECL and SBCL if you obtain the crashing behavior on your machine too?

Also, running an arm64 linux VM on M* MacOS can be another option in the meanwhile.

LEHunter commented 4 months ago

Well, this is interesting. I installed SBCL, and py4cl2-cffi loads without a hitch. I couldn’t run the tests though. I tried git clone https://github.com/digikar99/py4cl2-cffi-tests in the quicklsp local-distributions directory, and got the result at the end of this message. However, this suggests a problem with Allegro CL, not py4cl2-cffi. I’ll file a bug report with them and see if they have anything to say…

This is my system, btw: % uname -a Darwin MacBook-Air-3.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:19:22 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8112 arm64

Thanks for your help!

Larry

debugger invoked on a UIOP/LISP-BUILD:COMPILE-FILE-ERROR in thread

<THREAD "main thread" RUNNING {70055206B3}>:

COMPILE-FILE-ERROR while compiling #<CL-SOURCE-FILE "cl-environments" "src" "other/sbcl">

LEHunter commented 4 months ago

OK, as soon as I sent that, I noticed that the gcc commands for SBCL and ACL differ. The ACL one is missing the -lpython3.12 at the end. When I add that manually, it compiles. I think this must be configuration issue, and am investigating further…

LEHunter commented 4 months ago

More hints:

(software-type) "64-bit Apple macOS 12.0 (Apple Silicon)”

And it appears your code is looking for “Darwin”…

On Apr 27, 2024, at 2:53 PM, Larry Hunter @.***> wrote:

OK, as soon as I sent that, I noticed that the gcc commands for SBCL and ACL differ. The ACL one is missing the -lpython3.12 at the end. When I add that manually, it compiles. I think this must be configuration issue, and am investigating further…

LEHunter commented 4 months ago

Looking further… software-type is not exported from UIOP, but closer-common-lisp (a CLOS compatibility layer) does expose it. Possibly better to use (uiop:operating-system) which returns :macos on my machine…

Hope this is enough for a patch.

Larry

On Apr 27, 2024, at 3:00 PM, Larry Hunter @.***> wrote:

More hints:

(software-type) "64-bit Apple macOS 12.0 (Apple Silicon)”

And it appears your code is looking for “Darwin”…

On Apr 27, 2024, at 2:53 PM, Larry Hunter @.***> wrote:

OK, as soon as I sent that, I noticed that the gcc commands for SBCL and ACL differ. The ACL one is missing the -lpython3.12 at the end. When I add that manually, it compiles. I think this must be configuration issue, and am investigating further…

digikar99 commented 4 months ago

Okay, great! And thanks for your reports, I will issue the fixes soon.

Well, this is interesting. I installed SBCL, and py4cl2-cffi loads without a hitch. I couldn’t run the tests though. I tried git clone https://github.com/digikar99/py4cl2-cffi-tests in the quicklsp local-distributions directory, and got the result at the end of this message. However, this suggests a problem with Allegro CL, not py4cl2-cffi. I’ll file a bug report with them and see if they have anything to say…

Oh right, py4cl2-cffi-tests has a number of lisp dependencies that need more recent versions than the one's in quicklisp. Sidestepping them, you can manually compile the defpymodule forms in the py4cl2-cffi-tests/package.lisp file and see if it produces the appropriate lisp packages and functions. I will also work on modularizing the tests further to see if these dependencies can be separated out.

Until we get a new quicklisp dist, you can either use ultralisp or download the dependencies using download-dependencies. /path/to/dependencies can be either quicklisp local projects or a more temporary directory.

LEHunter commented 4 months ago

Great! There was an important typo in one of those emails. (Uiop:operating-system) returns :macosx. (Note x).

Didn’t know about ultralisp, I’ll give it a try when the fixes are ready.

Larry

On Apr 27, 2024, at 3:57 PM, Shubhamkar Ayare @.***> wrote:

Okay, great! And thanks for your reports, I will issue the fixes soon.

Well, this is interesting. I installed SBCL, and py4cl2-cffi loads without a hitch. I couldn’t run the tests though. I tried git clone https://github.com/digikar99/py4cl2-cffi-tests in the quicklsp local-distributions directory, and got the result at the end of this message. However, this suggests a problem with Allegro CL, not py4cl2-cffi. I’ll file a bug report with them and see if they have anything to say…

Oh right, py4cl2-cffi-tests has a number of lisp dependencies that need more recent versions than the one's in quicklisp. Sidestepping them, you can manually compile the defpymodule forms in the py4cl2-cffi-tests/package.lisp file and see if it produces the appropriate lisp packages and functions. I will also work on modularizing the tests further to see if these dependencies can be separated out.

Until we get a new quicklisp dist, you can either use ultralisp or download the dependencies using download-dependencies https://github.com/digikar99/download-dependencies. /path/to/dependencies can be either quicklisp local projects or a more temporary directory.

— Reply to this email directly, view it on GitHub https://github.com/digikar99/py4cl2-cffi/issues/13#issuecomment-2081289277, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACWZKKVEUD3DJH6BWYSI33Y7RJSPAVCNFSM6AAAAABGRYXONGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBRGI4DSMRXG4. You are receiving this because you authored the thread.

digikar99 commented 4 months ago

I couldn’t run the tests though. I tried git clone https://github.com/digikar99/py4cl2-cffi-tests in the quicklsp local-distributions directory, and got the result at the end of this message.

I have split the tests now. You should be able to (ql:quickload "py4cl2-cffi-tests-lite") against the current quicklisp dist. If you want to run the full tests, check out ultralisp or download-dependencies. I'll probably add clpm support some day.

(Uiop:operating-system) returns :macosx.

Same behavior on amd64 Mac. And thanks for pointing to this one! I have replaced (software-type) with (uiop:operating-system).

I just discovered -framework CoreFoundation flags from github actions' python3-config --libs and python3-config --ldflags. But, adding these flags didn't change anything --- github actions still crashes on M1 and freezes on non-M1. The non-M1 Mac I have access to passes the tests without crashing though!

digikar99 commented 4 months ago

Okay, a very simple cause and fix.

https://github.com/digikar99/py4cl2-cffi/commit/44f07842827350418d2337f5c34d8d8e3a62d5a7: Mask all the float traps while loading numpy.

Tests pass now! (This is on the macos branch. I will merge into master soon.)

LEHunter commented 4 months ago

Still trying to get the initial py4cl2-cffi macOS branch to work in my environment. Two things:

  1. Made a pull request to get ride of the quote in the numpy-installed-p file — it screws up case sensitive lower lisps (who don’t define T, only t).

  2. Still getting problems with the compile of py4cl-numpy-utils.c not finding the right (arm64) libraries. I think this is a configuration problem. The error looks like:

py4cl2-cffi> (may-be-compile-numpy-utils-shared-object) gcc @./Frameworks/Python.framework/Versions/3.12/include/python3.12 @./Frameworks/Python.framework/Versions/3.12/include/python3.12 -I'/opt/homebrew/lib/python3.12/site-packages/numpy/core/include/' -c -Wall -Werror -fpic py4cl-numpy-utils.c && gcc -shared -o libpy4cl-numpy-utils.so py4cl-numpy-utils.o Undefined symbols for architecture x86_64: "_PyCapsule_GetPointer", referenced from: …

A correct invocation would look like:

gcc @./Frameworks/Python.framework/Versions/3.12/include/python3.12 -I/opt/homebrew/lib/python3.12/site-packages/numpy/core/include/ -c -Wall -Werror -fpic py4cl-utils.c && gcc @./Frameworks/Python.framework/Versions/3.12/lib -shared -o libpy4cl-utils.so py4cl-utils.o -lpython3.12

Which works from the command line. Looks to me as if the library path is not being included properly (or at all). I couldn’t back out your configuration logic completely, so I don’t have a specific pull request yet. Also, I think your approach for naming looks like it should be generating libpy4cl-utils.dylib, not libpy4cl-utils.so — I’m not sure why that’s not working, and I might try a case statement instead of the object dispatch approach in shared-objects.lisp file. These might be useful in tracking this down:

py4cl2-cffi> python-ldflags @.***/Frameworks/Python.framework/Versions/3.12/lib/python3.12/config-3.12-darwin" "-lpython3.12" "-ldl" "-framework" "CoreFoundation”)

py4cl2-cffi> (shared-library-from-ldflag python-ldflags (intern (uiop:operating-system) :keyword)) "lib(-ldl -framework CoreFoundation).dylib”

Thanks again for all your help with this (and for writing it in the first place!)

Larry

digikar99 commented 4 months ago

Still getting problems with the compile of py4cl-numpy-utils.c not finding the right (arm64) libraries. I think this is a configuration problem.

Is the config-darwin.lisp actually loading? It should be performing the configuration correctly. Is non-NIL (member :darwin cl:*features*) on your system? In py4cl2-cffi.asd, "py4cl2-cffi" depends on "py4cl2-cffi/config-darwin" only when the :darwin feature is present. May be this should be changed?

Otherwise, I will make the dependence unconditional. And leave it up to the (uiop:operating-system) to handle it correctly at the end of config-darwin.lisp.

(when (eq :macosx (uiop:operating-system))
  (configure))

(shared-library-from-ldflag python-ldflags (intern (uiop:operating-system) :keyword))

load-python-and-libraries in shared-objects.lisp calls %shared-library-from-ldflag only when the flag name starts with -l. So, these errors should not be arising, could you confirm?

LEHunter commented 4 months ago

On Apr 29, 2024, at 6:09 PM, Shubhamkar Ayare @.***> wrote:

Still getting problems with the compile of py4cl-numpy-utils.c not finding the right (arm64) libraries. I think this is a configuration problem.

Is the config-darwin.lisp actually loading? It should be performing the configuration correctly. Is non-NIL (member :darwin cl:features) on your system? In py4cl2-cffi.asd, "py4cl2-cffi" depends on "py4cl2-cffi/config-darwin" only when the :darwin feature is present. May be this should be changed?

I believe config-darwin is loading.

cl-user> (member :darwin cl:features) (:darwin :global-vars :thread-support :swank :quicklisp :asdf3.3 :asdf3.2 :asdf3.1 :asdf3 :asdf2 :asdf :has-lockf :has-mkdtemp :acldns :os-macosx :os-unix :asdf-unicode :package-local-nicknames :allegro-cl-enterprise :rfc8141 :rfc6874 :rfc3986 :has-clos-fixed-index-feature :ipv6 :acl-socket :hiper-socket :profiler :|YACC Release 0.98| :compiler :use-structs-in-compiler :clos :atomic-subword-setf :atomic-setf :verify-car-cdr :target-mac :dynload :dlmac :unix :macosarm64 :macosx64 :macosx :macos :arm64 :jitcode :little-endian :encapsulating-efs :double-float-random-fix :relative-package-names :module-versions :ieee :ieee-floating-point :conforming-ieee :ics :common-lisp :ansi-cl :draft-ansi-cl-2 :x3j13 :allegro :excl :franz-inc :allegro-version>= :allegro-version= :new-environments :flavors :multiprocessing :use-thread-libs :allegro-v11.0 :fixnargs :64bit :smp-macros :ssl-support :thread-resident-symbol-trampoline) cl-user>

NB: I prefer FIND to MEMBER in these circumstances (less consing), but doesn’t really matter.

Otherwise, I will make the dependence unconditional. And leave it up to the (uiop:operating-system) to handle it correctly at the end of config-darwin.lisp.

(when (eq :macosx (uiop:operating-system)) (configure)) This does appear to be executing. NB: I think this (and any other lop level form) should be wrapped in a (eval-when (:compile-toplevel :load-toplevel :execute) … ) (shared-library-from-ldflag python-ldflags (intern (uiop:operating-system) :keyword))

load-python-and-libraries in shared-objects.lisp calls %shared-library-from-ldflag only when the flag name starts with -l. So, these errors should not be arising, could you confirm?

There may be a case problem here. Sometimes the flag is -L and sometime -l. Maybe that’s the problem?

Larry

digikar99 commented 4 months ago

This does appear to be executing. NB: I think this (and any other lop level form) should be wrapped in a (eval-when (:compile-toplevel :load-toplevel :execute) … )

Ah, this should be it!

There may be a case problem here. Sometimes the flag is -L and sometime -l. Maybe that’s the problem?

The -l flag is for the library, the -L flag is for the search paths.

digikar99 commented 4 months ago

This does appear to be executing. NB: I think this (and any other lop level form) should be wrapped in a (eval-when (:compile-toplevel :load-toplevel :execute) … )

Ah, this should be it!

But, may be no, compile and load of config-darwin.lisp should both take place before shared-objects.lisp itself. I will try but don't expect it to change anything.

digikar99 commented 4 months ago

I believe config-darwin is loading.

Could you manually evaluate (py4cl2-cffi/config-darwin::configure) to see if the configuration variables are incorrect. (I assume you are at the latest commit of macos branch.)

digikar99 commented 4 months ago

You might need to pull with the -f flag though.

LEHunter commented 4 months ago

On Apr 29, 2024, at 6:36 PM, Shubhamkar Ayare @.***> wrote:

I believe config-darwin is loading.

Could you manually evaluate (py4cl2-cffi/config-darwin::configure) to see if the configuration variables are incorrect. (I assume you are at the latest commit of macos branch.)

Maybe not, actually: :Darwin is on features but the config-darrwin file does not seem to be getting loaded…

local-projects % git clone https://github.com/digikar99/py4cl2-cffi --branch macos Cloning into 'py4cl2-cffi'... remote: Enumerating objects: 1348, done. remote: Counting objects: 100% (758/758), done. remote: Compressing objects: 100% (226/226), done. remote: Total 1348 (delta 569), reused 674 (delta 503), pack-reused 590 Receiving objects: 100% (1348/1348), 356.56 KiB | 1.36 MiB/s, done. Resolving deltas: 100% (872/872), done. local-projects % lisp International Allegro CL Enterprise Edition 11.0 [64-bit macOS (Apple Silicon)] Copyright (C) 1985-2023, Franz Inc., Lafayette, CA, USA. All Rights Reserved.

This development copy of Allegro CL is licensed to: [TC22861]

Note: Allegro CL will expire in 62 days, on June 30, 2024. ; Loading /Users/hunter/quicklisp/setup.lisp ; Fast loading ; /Applications/AllegroCL64.app/Contents/Resources/code/asdf.001 ;;; Installing asdf patch, version 1. ; Autoloading for class synonym-stream: ; Fast loading from bundle code/streamc.fasl. ; Fast loading from bundle code/efft-utf-8s-base.fasl. ; Fast loading from bundle code/efft-utf8-base.fasl. ; Fast loading from bundle code/efft-void.fasl. ; Fast loading from bundle code/efft-latin1-base.fasl. ; Autoloading for package "excl.osi": ; Fast loading ; /Applications/AllegroCL64.app/Contents/Resources/code/osi.fasl ; Fast loading from bundle code/fileutil.fasl. ; Fast loading from bundle code/acldns.fasl. ; Fast loading from bundle code/iodefs.fasl. ; Fast loading from bundle code/iordefs.fasl. ; Fast loading from bundle code/efmacs.fasl. ;; Optimization settings: safety 1, space 1, speed 1, debug 2, ;; compilation-speed 1. ;; For a complete description of all compiler switches given the ;; current optimization settings evaluate (explain-compiler-settings). ;;--- ;; Current reader case mode: :case-sensitive-lower cl-user(1): (ql:quickload "py4cl2-cffi") To load "py4cl2-cffi": Load 1 ASDF system: py4cl2-cffi ; Loading "py4cl2-cffi" [package bordeaux-threads]........................ [package bordeaux-threads-2]...................... [package py4cl2-cffi/config] ("Python" "3.12.3" "") @./Frameworks/Python.framework/Versions/3.12/lib/python3.12/config-3.12-darwin" "-lpython3.12" "-ldl" "-framework" "CoreFoundation" "") ("Python" "3.12.3" "") @./Frameworks/Python.framework/Versions/3.12/include/python3.12" @./Frameworks/Python.framework/Versions/3.12/include/python3.12" "") ("/opt/homebrew/bin/python" "") ("" @./3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python312.zip" @./3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12" @./3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload" "/opt/homebrew/lib/python3.12/site-packages" "") ...................... [package py4cl2-cffi]. gcc @./Frameworks/Python.framework/Versions/3.12/include/python3.12 @./Frameworks/Python.framework/Versions/3.12/include/python3.12 -c -Wall -Werror -fpic py4cl-utils.c && gcc -shared -o libpy4cl-utils.so py4cl-utils.o gcc @./Frameworks/Python.framework/Versions/3.12/include/python3.12 @./Frameworks/Python.framework/Versions/3.12/include/python3.12 -I'/opt/homebrew/lib/python3.12/site-packages/numpy/core/include/' -c -Wall -Werror -fpic py4cl-numpy-utils.c && gcc -shared -o libpy4cl-numpy-utils.so py4cl-numpy-utils.o Undefined symbols for architecture arm64: "_PyCapsule_GetPointer", referenced from: import_array in py4cl-numpy-utils.o "_PyCapsule_Type", referenced from: __import_array in py4cl-numpy-utils.o "_PyErr_Format", referenced from: import_array in py4cl-numpy-utils.o import_array in py4cl-numpy-utils.o "_PyErr_Print", referenced from: _import_numpy in py4cl-numpy-utils.o "_PyErr_SetString", referenced from: _import_numpy in py4cl-numpy-utils.o __import_array in py4cl-numpy-utils.o import_array in py4cl-numpy-utils.o import_array in py4cl-numpy-utils.o __import_array in py4cl-numpy-utils.o "_PyExc_ImportError", referenced from: _import_numpy in py4cl-numpy-utils.o "_PyExc_RuntimeError", referenced from: import_array in py4cl-numpy-utils.o "_PyImport_ImportModule", referenced from: import_array in py4cl-numpy-utils.o "_PyObject_GetAttrString", referenced from: __import_array in py4cl-numpy-utils.o "Py_Dealloc", referenced from: import_array in py4cl-numpy-utils.o __import_array in py4cl-numpy-utils.o import_array in py4cl-numpy-utils.o ld: symbol(s) not found for architecture arm64 clang: error: linker command failed with exit code 1 (use -v to see invocation) ; While evaluating an unknown function at top level: Error: Subprocess with command "gcc @./Frameworks/Python.framework/Versions/3.12/include/python3.12 @./Frameworks/Python.framework/Versions/3.12/include/python3.12 -I'/opt/homebrew/lib/python3.12/site-packages/numpy/core/include/' -c -Wall -Werror -fpic py4cl-numpy-utils.c && gcc -shared -o libpy4cl-numpy-utils.so py4cl-numpy-utils.o" exited with error code 1 [condition type: subprocess-error] Problem detected when processing (let* ...) inside (excl:compiler-let ...) inside (eval-when (:compile-toplevel :load-toplevel) ...)

Restart actions (select using :continue): 0: IGNORE-ERROR-STATUS 1: retry the compilation of /Users/hunter/quicklisp/local-projects/py4cl2-cffi/src/shared-objects.lisp 2: continue compiling /Users/hunter/quicklisp/local-projects/py4cl2-cffi/src/shared-objects.lisp but generate no output file 3: Retry compiling #<cl-source-file "py4cl2-cffi" "shared-objects">. 4: Continue, treating compiling #<cl-source-file "py4cl2-cffi" "shared-objects"> as having been successful. 5: Retry ASDF operation. 6: Retry ASDF operation after resetting the configuration. 7: Give up on "py4cl2-cffi" 8: Register local projects and try again. 9: Return to Top Level (an "abort" restart). 10: Abort entirely from this (lisp) process.

[changing package from "common-lisp-user" to "py4cl2-cffi"] [1c] py4cl2-cffi(2): (py4cl2-cffi/config-darwin::configure)

Error: Package "py4cl2-cffi/config-darwin" not found. [condition type: reader-error] Problem detected when processing (let* ...) inside (excl:compiler-let ...) inside (eval-when (:compile-toplevel :load-toplevel) ...)

Restart actions (select using :continue): 0: Return to Debug Level 1 (an "abort" restart). 1: IGNORE-ERROR-STATUS 2: retry the compilation of /Users/hunter/quicklisp/local-projects/py4cl2-cffi/src/shared-objects.lisp 3: continue compiling /Users/hunter/quicklisp/local-projects/py4cl2-cffi/src/shared-objects.lisp but generate no output file 4: Retry compiling #<cl-source-file "py4cl2-cffi" "shared-objects">. 5: Continue, treating compiling #<cl-source-file "py4cl2-cffi" "shared-objects"> as having been successful. 6: Retry ASDF operation. 7: Retry ASDF operation after resetting the configuration. 8: Give up on "py4cl2-cffi" 9: Register local projects and try again. 10: Return to Top Level (an "abort" restart). 11: Abort entirely from this (lisp) process. [2] py4cl2-cffi(3): (find :darwin cl:features) :darwin [2] py4cl2-cffi(4):

digikar99 commented 4 months ago

Maybe not, actually: :Darwin is on features but the config-darrwin file does not seem to be getting loaded…

Okay, with 58bfb349429d926e31cc665e9552fb28311bd819, config-darwin should load unconditionally now.

There are other errors now which I'm looking into. Between 61cc8ef93c908d91c3aecb6a39f837f469eeabbf and 53402f365b0b8cb8940525d2f6d63fffb66aa9a1, something changed which is leading to memory fault during (py4cl2-cffi:pystart) in the presence of numpy.

digikar99 commented 4 months ago

467122112bebdd288a768f4f1937f2dbfd125223 fixes the recent crashing.

Some changes to the tests are still required to handle the readtable case correctly. I will ping once those are ready. Although, py4cl2-cffi should itself still be useable.

LEHunter commented 4 months ago

On Apr 29, 2024, at 4:09 PM, Larry Hunter @.***> wrote:

@.***/Frameworks/Python.framework/Versions/3.12/lib -shared -o libpy4cl-utils.so http://libpy4cl-utils.so/ py4cl-utils.o -lpython3.12

Regardless, the the call to compile py4cl-numpy-utils.c is not getting any load flags, -l or -L in my environment now.

digikar99 commented 4 months ago

But py4cl-utils.c does get the appropriate load flags?

LEHunter commented 4 months ago

But py4cl-utils.c does get the appropriate load flags?

No, but it doesn’t appear to need the ones it’s not getting. Here are the invocations of each:

cl-user(1): (ql:quickload "py4cl2-cffi") To load "py4cl2-cffi": Load 1 ASDF system: py4cl2-cffi ; Loading "py4cl2-cffi" [package bordeaux-threads]........................ [package bordeaux-threads-2]...................... [package py4cl2-cffi/config] ("Python" "3.12.3" "") @./Frameworks/Python.framework/Versions/3.12/lib/python3.12/config-3.12-darwin" "-lpython3.12" "-ldl" "-framework" "CoreFoundation" "") ("Python" "3.12.3" "") @./Frameworks/Python.framework/Versions/3.12/include/python3.12" @./Frameworks/Python.framework/Versions/3.12/include/python3.12" "") ("/opt/homebrew/bin/python" "") ("" @./3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python312.zip" @./3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12" @./3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload" "/opt/homebrew/lib/python3.12/site-packages" "") ...................... [package py4cl2-cffi]. gcc @./Frameworks/Python.framework/Versions/3.12/include/python3.12 @./Frameworks/Python.framework/Versions/3.12/include/python3.12 -c -Wall -Werror -fpic py4cl-utils.c && gcc -shared -o libpy4cl-utils.so py4cl-utils.o gcc @./Frameworks/Python.framework/Versions/3.12/include/python3.12 @./Frameworks/Python.framework/Versions/3.12/include/python3.12 -I'/opt/homebrew/lib/python3.12/site-packages/numpy/core/include/' -c -Wall -Werror -fpic py4cl-numpy-utils.c && gcc -shared -o libpy4cl-numpy-utils.so py4cl-numpy-utils.o Undefined symbols for architecture arm64: "_PyCapsule_GetPointer", referenced from: __import_array in py4cl-numpy-utils.o

digikar99 commented 4 months ago

It appears that the (py4cl2-cffi/config-darwin::configure) function is still not being called.

Let me know the output of this:

(in-package :py4cl2-cffi/config-darwin)
(when (print (eq :macosx (print (uiop:operating-system))))
  (configure))
LEHunter commented 4 months ago

Oh right, I haven’t download yesterday’s fixes…. Hold on. I have some meetings coming up, and won't be able to get back to this for a couple of hours. I’ll do a git update and try again then.

On Apr 30, 2024, at 7:22 AM, Shubhamkar Ayare @.***> wrote:

It appears that the (py4cl2-cffi/config-darwin::configure) function is still not being called.

Let me know the output of this:

(in-package :py4cl2-cffi/config-darwin) (when (print (eq :macosx (print (uiop:operating-system)))) (configure)) — Reply to this email directly, view it on GitHub https://github.com/digikar99/py4cl2-cffi/issues/13#issuecomment-2086102372, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACWZKMD7BNBS3OD5KXQ7U3Y77HNVAVCNFSM6AAAAABGRYXONGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBWGEYDEMZXGI. You are receiving this because you authored the thread.

digikar99 commented 4 months ago

Alright, test it whenever you are free next. Hoping it works for you!

LEHunter commented 4 months ago

The good news is the latest update fixes the numpy-utils compilation problem! I still get a warning when compiling numpy-utils, but it seems to work: ld: warning: ignoring file @.***/3.12.3/Frameworks/Python.framework/Versions/3.12/Python': found architecture 'arm64', required architecture 'x86_64'

Bad news is we are not done yet.

Next problem is

Error: Attempt to take the value of the unbound variable `+python-function-reference-type-alist+'. [condition type: unbound-variable] Problem detected when processing (pyforeign-funcall "Py_IncRef" :pointer pyobject-pointer)

when compiling. That’s because you evaluate it when defining the macro pyforeign-funcall. Fix this by wrapping the defconstant with eval-when:

(eval-when (:load-toplevel :compile-toplevel :execute)

(define-constant +python-function-reference-type-alist+ '(("Py_Initialize" nil) ... ("PY4CL_PyArray_FromArray" :new)) :test #'equal))

After fixing that, I get

While file-compiling

'(:top-level-form

  "/Users/hunter/quicklisp/local-projects/py4cl2-cffi/src/pythonizers.lisp"
  2494)

in

P"/Users/hunter/quicklisp/local-projects/py4cl2-cffi/src/pythonizers.lisp"

; starting at file character position 2494: Error: t fell through a etypecase form. The valid cases were (eql float-features::T) and list. [condition type: case-failure]

This is a bug in float-features (where there’s a T instead of a t in the definition of the macro with-float-traps-masked. I filed a pull request to that repo.

Then, I got an error when loading the compiled pythonizers file. This one still mystifies me…

(ql:quickload "py4cl2-cffi")

To load "py4cl2-cffi":

Load 1 ASDF system:

py4cl2-cffi

; Loading "py4cl2-cffi"

[package bordeaux-threads]........................

[package bordeaux-threads-2]......

("Python" "3.12.3" "")

@.***/Frameworks/Python.framework/Versions/3.12/lib/python3.12/config-3.12-darwin"

"-lpython3.12" "-ldl" "-framework" "CoreFoundation" "")

@.***/Frameworks/Python.framework/Versions/3.12/include/python3.12"

@.***/Frameworks/Python.framework/Versions/3.12/include/python3.12"

"")

("/opt/homebrew/bin/python" "")

(""

@.***/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python312.zip"

@.***/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12"

@.***/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload"

"/opt/homebrew/lib/python3.12/site-packages" "") ................

[package py4cl2-cffi]..

gcc @./Frameworks/Python.framework/Versions/3.12/include/python3.12 @./Frameworks/Python.framework/Versions/3.12/include/python3.12 -c -Wall -Werror -fpic py4cl-utils.c && gcc @.***/Frameworks/Python.framework/Versions/3.12//lib' -framework CoreFoundation -shared -o libpy4cl-utils.so py4cl-utils.o -lpython3.12

......

Error: No class named: foreign-pointer.

[condition type: program-error]

It seems to have a type definition:

py4cl2-cffi> (describe 'foreign-pointer) foreign-pointer is a tenured symbol. It is unbound. It is external in the cffi-sys package and exported from the cffi package and accessible in the py4cl2-cffi package. Its property list has these indicator/value pairs: excl::deftype-expander #<Function (excl::deftype-expander foreign-pointer)> ; No value

But calling

py4cl2-cffi> (find-class 'foreign-pointer)

Generates the No class named…error. Ideas?

Larry

digikar99 commented 4 months ago

(find-class 'foreign-pointer)

pythonize defined in pythonizers.lisp is a generic function. One of its methods dispatches over the class corresponding to foreign-pointer. On SBCL, this is sb-sys:system-area-pointer. Could you find an equivalent class on Allegro CL? typexpand or type-expand or equivalent should be useful for this, or the Allegro CL documentation, or their support channels.

digikar99 commented 4 months ago

Otherwise, I could merge the pythonize method dispatching over foreign-pointer into its default method. This would be a much portable solution.

LEHunter commented 4 months ago

I believe in ACL it would be ff:foreign-pointer

(find-class 'ff:foreign-pointer)

See https://franz.com/support/documentation/10.1/doc/foreign-functions.htm

Larry

On Apr 30, 2024, at 6:09 PM, Shubhamkar Ayare @.***> wrote:

(find-class 'foreign-pointer)

pythonize defined in pythonizers.lisp is a generic function. One of its methods dispatches over the class corresponding to foreign-pointer. On SBCL, this is sb-sys:system-area-pointer. Could you find an equivalent class on Allegro CL? typexpand or type-expand or equivalent should be useful for this, or the Allegro CL documentation, or their support channels.

— Reply to this email directly, view it on GitHub https://github.com/digikar99/py4cl2-cffi/issues/13#issuecomment-2087946435, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACWZKNBAWKPJUQBD67SQU3ZABTI7AVCNFSM6AAAAABGRYXONGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBXHE2DMNBTGU. You are receiving this because you authored the thread.

LEHunter commented 4 months ago

Sorry, outdated documentation pointer. Current version (largely the same) is here https://franz.com/support/documentation/11.0/foreign-functions.html

On Apr 30, 2024, at 3:40 PM, Larry Hunter @.***> wrote:

The good news is the latest update fixes the numpy-utils compilation problem! I still get a warning when compiling numpy-utils, but it seems to work: ld: warning: ignoring file @.***/3.12.3/Frameworks/Python.framework/Versions/3.12/Python': found architecture 'arm64', required architecture 'x86_64'

Bad news is we are not done yet.

Next problem is

Error: Attempt to take the value of the unbound variable `+python-function-reference-type-alist+'. [condition type: unbound-variable] Problem detected when processing (pyforeign-funcall "Py_IncRef" :pointer pyobject-pointer)

when compiling. That’s because you evaluate it when defining the macro pyforeign-funcall. Fix this by wrapping the defconstant with eval-when:

(eval-when (:load-toplevel :compile-toplevel :execute)

(define-constant +python-function-reference-type-alist+ '(("Py_Initialize" nil) ... ("PY4CL_PyArray_FromArray" :new)) :test #'equal))

After fixing that, I get

While file-compiling

'(:top-level-form

  "/Users/hunter/quicklisp/local-projects/py4cl2-cffi/src/pythonizers.lisp"
  2494)

in

P"/Users/hunter/quicklisp/local-projects/py4cl2-cffi/src/pythonizers.lisp"

; starting at file character position 2494: Error: t fell through a etypecase form. The valid cases were (eql float-features::T) and list. [condition type: case-failure]

This is a bug in float-features (where there’s a T instead of a t in the definition of the macro with-float-traps-masked. I filed a pull request to that repo.

Then, I got an error when loading the compiled pythonizers file. This one still mystifies me…

(ql:quickload "py4cl2-cffi")

To load "py4cl2-cffi":

Load 1 ASDF system:

py4cl2-cffi

; Loading "py4cl2-cffi"

[package bordeaux-threads]........................

[package bordeaux-threads-2]......

("Python" "3.12.3" "")

@.***/Frameworks/Python.framework/Versions/3.12/lib/python3.12/config-3.12-darwin"

"-lpython3.12" "-ldl" "-framework" "CoreFoundation" "")

@.***/Frameworks/Python.framework/Versions/3.12/include/python3.12"

@.***/Frameworks/Python.framework/Versions/3.12/include/python3.12"

"")

("/opt/homebrew/bin/python" "")

(""

@.***/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python312.zip"

@.***/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12"

@.***/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload"

"/opt/homebrew/lib/python3.12/site-packages" "") ................

[package py4cl2-cffi]..

gcc @./Frameworks/Python.framework/Versions/3.12/include/python3.12 @./Frameworks/Python.framework/Versions/3.12/include/python3.12 -c -Wall -Werror -fpic py4cl-utils.c && gcc @.***/Frameworks/Python.framework/Versions/3.12//lib' -framework CoreFoundation -shared -o libpy4cl-utils.so py4cl-utils.o -lpython3.12

......

Error: No class named: foreign-pointer.

[condition type: program-error]

It seems to have a type definition:

py4cl2-cffi> (describe 'foreign-pointer) foreign-pointer is a tenured symbol. It is unbound. It is external in the cffi-sys package and exported from the cffi package and accessible in the py4cl2-cffi package. Its property list has these indicator/value pairs: excl::deftype-expander #<Function (excl::deftype-expander foreign-pointer)> ; No value

But calling

py4cl2-cffi> (find-class 'foreign-pointer)

Generates the No class named…error. Ideas?

Larry

digikar99 commented 4 months ago

I still get a warning when compiling numpy-utils

This still remains.

Fix this by wrapping the defconstant with eval-when:

(eval-when (:load-toplevel :compile-toplevel :execute) ...)

Done.

I could merge the pythonize method dispatching over foreign-pointer into its default method. This would be a much portable solution.

Done.

I have pushed these changes to the macos branch. Let me know if this works.

digikar99 commented 4 months ago

I think there might be other places which require wrapping in eval-when too. I will check with asdf:compile-system and get back.

LEHunter commented 4 months ago

Possibly. This should usually just be for top level forms that are not definitions (e.g. start def*). However, you also have to watch out for things used by macros (which evaluate at unusual times). I think there are a couple of eval-when wrappers you could get rid of….

On Apr 30, 2024, at 7:01 PM, Shubhamkar Ayare @.***> wrote:

I think there might be other places which require wrapping in eval-when too. I will check with asdf:compile-system and get back.

— Reply to this email directly, view it on GitHub https://github.com/digikar99/py4cl2-cffi/issues/13#issuecomment-2087980701, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACWZKPKKYRE4JFDSHAXVHTZABZLFAVCNFSM6AAAAABGRYXONGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBXHE4DANZQGE. You are receiving this because you authored the thread.

LEHunter commented 4 months ago

Turns out the maintainer of float-features is not going to fix the bug…

Begin forwarded message:

From: Yukari Hafner @.> Subject: Re: [Shinmera/float-features] Fixed bug for case-sensitive lisp readers (PR #35) Date: April 30, 2024 at 8:35:21 PM HST To: Shinmera/float-features @.> Cc: Larry Hunter @.>, Author @.> Reply-To: Shinmera/float-features @.***>

I've rejected all of these kinds of "fixes" for all of my libraries and will continue to do so. I have good reason to capitalise constants in code, and I'm not going to stop doing it. It is not my responsibility to cater my code to your customisations to the default environment. Just the same as I'm not going to account for changes to read-base even if it is "ANSI standard", I'm not going to account for changes to the read case.

digikar99 commented 4 months ago

Turns out their license is permissive enough to adapt their code to our needs with appropriate acknowledgements.

LEHunter commented 4 months ago

OK, making some progress. The current macOS branch of py4cl2-cffi quickloads. After fixing case-mode bugs in clunit and iterate, I’ve gotten the tests-lite to load. But it fails immediately because calls to pystart fail with "attempt to call `system::ff-funcall' which is an undefined function.” This in turn comes from mkfifo, which calls cffi:foreign-function At this point, I am stalled. Ideas?

Larry

digikar99 commented 4 months ago

I wonder what an appropriate configuration for ASDF would be to let it use the case insensitive readtable by default, and any other custom readtable only when a user explicitly specifies so.

I have also wanted package local readtables, but I don't see any good way other that dirty patching the lisp system to redefine certain core functions like read and compile. Or creating a shadowing CL package which acknowledges this. Well, I guess I can include that in peltadot - I already have a shadowing CL package there, including a shadowing compile function.

About the system::ff-funcall, can you check if it's possible to cffi:foreign-funcall for any other foreign function? If that does not work, may be CFFI might need to be updated for Allegro CL. But I doubt it could be as grave as that! Certainly, other Allegro CL users must be using CFFI, so it should be up to date.

PS: It will be another 2-3 days before I continue development. Some non-lisp tasks require my attention.

LEHunter commented 4 months ago

Interesting. I think it’s mostly a matter of testing, rather than trying for package-local read tables (which seem heavy duty, but doable). The problems in clunit and iterate were simple and easy to find (using all caps in case-insensitive lisps is a dead giveaway). I don’t mind living with local installs of these that I fix myself, although I will submit pull requests to the repositories and see if they are more open minded than the float-features guy.

I haven’t started looking at cffi — this looks like it might be a simple config issue. I can poke around for the next few days and see what I find. I’m also doing this as a side project, I totally understand.

On May 1, 2024, at 4:54 PM, Shubhamkar Ayare @.***> wrote:

I wonder what an appropriate configuration for ASDF would be to let it use the case insensitive readtable by default, and any other custom readtable only when a user explicitly specifies so.

I have also wanted package local readtables, but I don't see any good way other that dirty patching the lisp system to redefine certain core functions like read and compile. Or creating a shadowing CL package which acknowledges this. Well, I guess I can include that in peltadot https://gitlab.com/digikar/peltadot - I already have a shadowing CL package there, including a shadowing compile function.

About the system::ff-funcall, can you check if it's possible to cffi:foreign-funcall for any other foreign function? If that does not work, may be CFFI might need to be updated for Allegro CL. But I doubt it could be as grave as that! Certainly, other Allegro CL users must be using CFFI, so it should be up to date.

PS: It will be another 2-3 days before I continue development. Some non-lisp tasks require my attention.

— Reply to this email directly, view it on GitHub https://github.com/digikar99/py4cl2-cffi/issues/13#issuecomment-2089441108, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACWZKKJARW3WQTSMUEHWX3ZAGTH3AVCNFSM6AAAAABGRYXONGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBZGQ2DCMJQHA. You are receiving this because you authored the thread.

LEHunter commented 4 months ago

Looking more into cffi, and it does appear to be broken on Allegro 11, and it’s not a case issue. I’ve come up with a fix (I wasn’t the only person who had a problem, see https://github.com/cffi/cffi/issues/381

But now we are back to the problem of apparently getting the wrong architecture somewhere in building libpy4cl-utils. I’m calling it a day, and will investigate further soon.

Unable to load foreign library (LIBPY4CL-UTILS.SO-80577). Loading /Users/hunter/quicklisp/local-projects/py4cl2-cffi/src/libpy4cl-utils.so failed with error: dlopen(/Users/hunter/quicklisp/local-projects/py4cl2-cffi/src/libpy4cl-utils.so, 0x000A): tried: @.***/lib/libpy4cl-utils.so' (no such file), '/Users/hunter/quicklisp/local-projects/py4cl2-cffi/src/libpy4cl-utils.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/hunter/quicklisp/local-projects/py4cl2-cffi/src/libpy4cl-utils.so' (no such file), '/Users/hunter/quicklisp/local-projects/py4cl2-cffi/src/libpy4cl-utils.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64')). [Condition of type cffi:load-foreign-library-error]