Could you please improve documentation (for beginners?)

Hello there, I'm trying to understand how to wrap a C library and I'm struggling. I got as far as calling c-include (both in the REPL and in a project that I'm building to contain the bindings) and I don't seem to know what to do now.

This is what you have currently:

This is a new c2ffi-based wrapper generator for Common Lisp with a focus, performance, convenience, and completeness. It works like this: (c-include "file.h") Tat's it. This calls c2ffi and generates architecture-specific .spec files you can distribute with your project.

After reading the entire readme, it feels like there's some knowledge missing. I generate these spec files, but I don't have any idea how to use them. Reading the readme gives me the impression that this was going to literally generate .lisp files binding C functions to low-level lisp functions. I was expecting that, after this was completed, it would then be a job of testing these functions out and tweaking them using the remaining documentation if I found issues.

I really can't see anything though; most of the readme is about all the different ways you use c-include. I'm sorry that this seems like I'm complaining - I'm not, I'm just a beginner trying to understand how this all works and how to achieve my goals. I don't know much about FFI and I'm trying my best to understand the process - perhaps a simple link to a relevant guide would be useful after the c-include mention. I don't understand how to actually use cl-autowrap to write bindings for a library and it is a little frustrating.

Thanks for reading. In case your interested, I'm trying to make some bindings for Raylib since the ones currently on GitHub here don't seem finished and development seems to have stalled. I'm not looking to make a replacement as I'm just a beginner, but if I were to get bindings that are usable enough to start a Raylib project with common lisp then maybe I'd consider uploading them as a temporary solution until someone who actually knows what they're doing would write better ones.

Reading the readme gives me the impression that this was going to literally generate .lisp files binding C functions to low-level lisp functions. I was expecting that, after this was completed, it would then be a job of testing these functions out and tweaking them using the remaining documentation if I found issues.

This is sortof the process, but, instead of generating a .lisp file, the .lisp file with the c-include gets compiled and that's the generated stuff. You can "see" what gets generated by looking either at the package contents (e.g. via slime/sly) or macroexpanding the c-include. (This seems a bit annoying, but vastly reduces the size and compile time, and streamlines the process.)

Note especially the bit about putting the c-include in its own separate package; it can really help here.

I got as far as calling c-include (both in the REPL and in a project that I'm building to contain the bindings) and I don't seem to know what to do now.

Getting this far is good; just make sure to check the output closely. If you're getting no definitions, or missing a lot of definitions, it's likely because c2ffi can't find headers (system, usually) in any of the predefined places. This is usually the big issue. Running c2ffi manually on the header in question and checking the output can help, as can writing a .h that sets up a few things then #includes the "real" one.

I would suggest building your own tiny library and making a wrapper for that first if you're not, or even something simple like a manually-declared int puts(char*);. Don't hesitate to jump into the lisp code either and see how foreign functions are called, or foreign types referenced, etc.

You might check ZMQ4L which is sortof a "small but more realistic" example, though I'm not entirely sure its current state vs the latest autowrap (or ZMQ for that matter; does anyone still use that?). cl-sdl2 was a "bigger" realistic example.

(Beyond this, I can try to answer specific questions or "what exactly do I do here" but I don't really have a ton of time for writing more extensive tutorials at this point.)

I really value your kind and in-depth answer to what even I would consider a pretty vague set of questions so thank you so much!

I guess that was the key takeaway, because I didn't really see an 'error' in the output of c-include, but you're right, the information it is not an indication of success either.

* (ql:quickload :raylisp)
To load "raylisp":
  Load 1 ASDF system:
    raylisp
; Loading "raylisp"
......; Note: skipping alias of type RAYLISP/AUTOWRAP::__BUILTIN_VA_LIST to
; RAYLISP/AUTOWRAP::VA-LIST due to undefined foreign type: __BUILTIN_VA_LIST
; Note: skipping alias of type RAYLISP/AUTOWRAP::__BUILTIN_VA_LIST to
; RAYLISP/AUTOWRAP::__GNUC_VA_LIST due to undefined foreign type:
; __BUILTIN_VA_LIST
............................................
..................................................
..................................................
..................; Total of 2 compile-time skipped definitions:
;   VA-LIST __GNUC_VA_LIST 
; Total of 1 compile-time missing entities:
;   RAYLISP/AUTOWRAP::__BUILTIN_VA_LIST 
; Total of 2 load-time skipped definitions:
;   VA-LIST __GNUC_VA_LIST 
; Total of 1 load-time missing entities:
;   RAYLISP/AUTOWRAP::__BUILTIN_VA_LIST 
; Note: skipping alias of type RAYLISP/AUTOWRAP::__BUILTIN_VA_LIST to
; RAYLISP/AUTOWRAP:VA-LIST due to undefined foreign type: __BUILTIN_VA_LIST
; Note: skipping alias of type RAYLISP/AUTOWRAP::__BUILTIN_VA_LIST to
; RAYLISP/AUTOWRAP:__GNUC_VA_LIST due to undefined foreign type:
; __BUILTIN_VA_LIST
; Total of 2 load-time skipped definitions:
;   VA-LIST __GNUC_VA_LIST 
; Total of 1 load-time missing entities:
;   RAYLISP/AUTOWRAP::__BUILTIN_VA_LIST

When I first saw the ... I thought It's working! But, unfortunately as I now know, I've been given the information I need in that output to fix the issue. Thank you for giving me some more information - my plan was initially to 'generate' these bindings and then write higher-level wrappers to wrap the low-level bindings this might generate, but seems like I've got a way to go before I can get to that point!

I'll keep trying my best. This is all I do when I try to invoke c-include:

;;;; raylisp.lisp

(in-package #:raylisp)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;; CL-Autowrap
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(eval-when (:compile-toplevel :load-toplevel :execute)
  (setf autowrap:*c2ffi-program* "~/Documents/c2ffi/build/bin/c2ffi")  
  (defvar *spec-path* (asdf:system-relative-pathname :raylisp "spec/"))
  (defvar *raylib-path* "~/Documents/raylib/src/raylib.h")
  (c-include (merge-pathnames *raylib-path* *spec-path*)
             :spec-path *spec-path*
             :definition-package :raylisp/autowrap
             :function-package :raylisp/autowrap
             :wrapper-package :raylisp/autowrap
             :accessor-package :raylisp/autowrap
             :constant-package :raylisp/autowrap
             :extern-package :raylisp/autowrap))

Nah that actually looks like pretty good output. (I don't recall how or if varargs stuff is handled... probably you just want to exclude the builtins to clean up the import at some point.)

There don't appear to be a ton of functions getting skipped. You should have useful symbols now in the raylisp/autowrap package. (Of course if you don't, then check the .spec files / output of c2ffi to make sure they're not missing; if they are, errors from c2ffi's output should help.)

At this point you should have some useful function you can call to verify it works. (Of course, with graphics stuff, this tends to have a lot of "main thread" interaction issues, so beware there.)

Nah that actually looks like pretty good output. (I don't recall how or if varargs stuff is handled... probably you just want to exclude the builtins to clean up the import at some point.)

There don't appear to be a ton of functions getting skipped. You should have useful symbols now in the raylisp/autowrap package. (Of course if you don't, then check the .spec files / output of c2ffi to make sure they're not missing; if they are, errors from c2ffi's output should help.)

At this point you should have some useful function you can call to verify it works. (Of course, with graphics stuff, this tends to have a lot of "main thread" interaction issues, so beware there.)

You say there's something inside the raylisp/autowrap package, but can I just ask, are we saying that something has written to a file, or are we saying that this is like.. symbols in the REPL? Are you suggesting I could call one of the C functions from lisp providing I know what its name and arguments are?

There should be symbols interned in raylisp/autowrap now that are (among other things) bound to functions; you should be able to call these from the repl, yes.

(If you're not using something like slime or sly, this is going to make your life a lot tougher; working with and analyzing the running lisp is an assumed part of this process, and the raw repl usually isn't the nicest. OTOH, threading...)

E.g., you should have something like RAYLISP/AUTOWRAP:INIT-WINDOW, and be able to do something like

(raylisp/autowrap:init-window 100 100 "foo")

...though depending on your thread situation this may or may not crash (not sure how raylib works). The function existing would be a good sign though.

There should be symbols interned in raylisp/autowrap now that are (among other things) bound to functions; you should be able to call these from the repl, yes.

(If you're not using something like slime or sly, this is going to make your life a lot tougher; working with and analyzing the running lisp is an assumed part of this process, and the raw repl usually isn't the nicest. OTOH, threading...)

E.g., you should have something like RAYLISP/AUTOWRAP:INIT-WINDOW, and be able to do something like
(raylisp/autowrap:init-window 100 100 "foo")
...though depending on your thread situation this may or may not crash (not sure how raylib works). The function existing would be a good sign though.

You're making me excited, I didn't realise I had access to these functions! I'm still learning lisp and since my occupation is a game developer I tend to use games as my way of learning something new. I wanted to try raylib and so I just needed some basic functions available in lisp; looking forward to getting them to finally work!

This is more than enough information for me to keep going - thank you! I'll close this issue since it's unfair for your project to be stained with me just asking questions :)

rpav / cl-autowrap

Could you please improve documentation (for beginners?) #109