C -> Scheme callbacks

[eval-apply]

Firstly, thanks for cyclone, its shaping up to be a very impressive scheme! I'm just getting to know cyclone (which so far is going great) so apologies if I've missed it, but I'm wondering if calling scheme from c is something that the current ffi is capable of? I'm trying to integrate cyclone with some libraries where it would be a useful pattern. Short of this is it possible to set a scheme global from c in a way that's thread safe?

[justinethier]

Thanks @eval-apply and I appreciate your interest in Cyclone!

This is an excellent question. The bad news is that we do not support calls from C to Scheme at this time. But as the tooling continues to evolve I would like to add support. This is somewhat complicated due to the Cyclone internals, with continuation-passing-style function chains allocating variables using the C stack.

Anyway, I think it is possible to set a Scheme global from C, but it would have to be done very carefully. Some things to keep in mind:

• You will need to use a mutex to lock the variable on the Scheme side before it is accessed, and you will need to use the same mutex on the C side to lock the variable before modifying it
• Globals are passed to our mutator functions as object *, and we reassign using *(glo) = value. See https://github.com/justinethier/cyclone/blob/master/runtime.c#L347
• The above should work as that single-line assignment if value is either an immediate object (boolean, char, integer) or an object with no children (double, string, bytevector, opaque, etc) you are allocating via malloc and managing independently of the Cyclone heap.

The above is also assuming you have a C thread running independently to a Scheme thread. It might be more flexible to suspend a Scheme thread and have it execute C code for awhile (indefinitely?). But more thought needs to be given to how that would work in practice. Is there a specific use case you have in mind? That might help drive the discussion.

Does that help? Please let me know if (when? :)) you have follow-up questions and I will do my best to point you in the right direction.

[eval-apply]

Thanks for the detailed reply, I was wondering if the compilation strategy would complicate matters.

At first, I'm largely only interested numeric values and perhaps bytevectors that I'd manage from the C side of things so that sounds fine with me. Scheme callbacks would be ideal but its certainly not a deal breaker.

To give you some more context, I've been prototyping some audio effects / musical instruments and have a hybrid environment where I can use scheme or C to both prototype and interact with the process as I've found that scheme is a fantastic environment for prototyping (unlike the industry standard C++).

So I'm running an audio loop in a separate thread that is established from C and from which I'd like to call scheme code (or get/set globals that are scheme visible). I have parts of this running in various scheme implementations already, but for a few reasons a migration to Cyclone really appeals to me (native threads, compilation, DLG GC and ER macros specifically are all winning elements).

[justinethier]

Well, what you are laying out sounds technically feasible. Please let me know if I can help out with anything, and/or if you have questions on the C integration.

That said, it would be best if Scheme calls could be made directly. No promises on when this will be implemented, but here are some ideas to start thinking about how this could work:

To make this happen we would need to setup a calling context via setjmp so we can longjmp when/if the stack is exhausted. A new gc_thread_data object is also required for record keeping purposes. I think this is do-able, though there seems like a lot of overhead if the intent is to make many short Scheme calls from C. Presumably that would be the common use case for a C thread. Otherwise it would make more sense to just run a Scheme thread and make quick calls out to C as needed.

There is also the issue that the way our Scheme is setup, C functions never return but rather just keep calling into their continuations forever. I wonder, what if we called into Scheme such that we entered with something like:

(lambda (return)
 ...
)

then the Scheme code could call (return obj) to pass the result back to C.

Certainly open to suggestions here and any comments regarding assumptions made above.

[justinethier]

@eval-apply You may find this useful, here is an example application demonstrating how to call a Scheme function from C:

https://github.com/justinethier/cyclone/tree/master/examples/callbacks

There are limitations as the Scheme function is not running in a fully-functional environment with GC, but it will work fine as long as just a handful of functions are called before returning to C. Memory will not be leaked as any new objects will be allocated locally on the stack.

Also, in c_trampoline what is happening is that the Scheme functions allocate data on the stack which we then return from using longjmp. That means any data pointed to by the returned object is still further up on the stack, so we need to either:

• Return an immediate value (number, boolean, char)
• Copy the object to a new object so the data is not overwritten by subsequent C calls (string, bytevector, etc)

Does that make sense? Happy to answer questions if you have any...

[eval-apply]

Fantastic! Thank you very much, hopefully I'll find some time shortly to continue this but this example you have setup is excellent. I'll try and apply it to what I'm working on here.

[justinethier]

@eval-apply No problem. I made a few minor tweaks to the example if you're interested, but no functional changes. Please let me know if there is anything I can do to help.

Let me see what it will take to put together another example with more complex Scheme calls. Assuming there are no deal breakers with our GC, such an example would allow more types of calls to be made, but would probably come with the caveat that the calls will be more expensive.

[justinethier]

@eval-apply - The examples under https://github.com/justinethier/cyclone/tree/master/examples/call-scm-from-c have been updated to include both a basic example as well as one that allows full execution of Scheme, including returning function callbacks.

You will need to update to the latest version of Cyclone (either master or the new 0.20 release) in order to build the examples and use the new calling interface.

[eval-apply]

Thanks so much for this Justin! When I opened the issue I really didn't expect that you'd develop a solution so quickly! I've got the examples working with 0.20 and I'm looking at ways to make use of Cyclone with my code base. Thanks again!

[justinethier]

Glad to help, James!