Invoke Lox callback/closure from native C code/library.

[HallofFamer]

I'm trying to add native functions/methods to the standard library of my own CLox implementation. Everything is going smoothly, until I come across native functions/methods that accept callback closures as argument.

The issue is that I will need to find a way to call the closure in native C code, but it seems extremely tricky to get this right. What I've tried is to make the vm function call(ObjClosure* closure, int argCount) public, and call this function with the closure argument. It does not seem to work, and the worse part is that it creates one more stack frame each time it is called so it limits how many times the closure may be invoked(no more than FRAME_MAX limit). Also I cant seem to get the return value from the closure, which is frustrating if the closure is a predicate that returns boolean value. Note this is actually very easy to do with JLox, all you have to do is to call LoxFunction.call(interpreter, arguments).

I wonder, has anyone actually tried to create a standard library for Lox? If so, what can you do about a native function/method that needs to accept a closure as argument and invoke it? I've browsed through all the CLox implementations and I cant find any examples, maybe it is a limitation of how CLox VM works?

[chrisjbreisch]

I am working on my own standard library, but I haven't yet stumbled upon a need for closures in it. I think I will, though. Your post is disappointing, as I think what you've described is pretty close to how I would have attempted it. I'm on a short break from my language, but I'll try to tinker with this soon and let you know if I come up with anything.

[HallofFamer]

@chrisjbreisch

According to what I heard from Wren's devs, the VM is not 'reentrant' and invoking callback from native C code is impossible without a major rewrite of the VM. Considering Lox VM is based on Wren VM, I fear the same can be said for Lox and it will not be possible to invoke Lox callback in C without a big change made on the Lox VM itself.

But please have a try yourself and see if you can come out with something. I am completely out of idea, and for now I just choose a hybrid approach to create any methods that invoke callbacks in Lox instead.

[RevengerWizard]

I'm kinda struggling onto this, mostly for functions on object that require a callback with different number of arguments to call. The problem with re-entrant VMs seems to be a common problem with different interpreters, at least as I've seen with Wren and Lua (and mine, as it's based on Lox, after all). Simpler way would be as Lua does it, first creating a C API for the language and then using that to handle the callback calls and for making the standard library.

[valkarias]

afaik its not reentrant because the cached variables of the VM loop need to be updated, one way to solve the issue is to remove these cached variables, taking a performance hit.

[HallofFamer]

@valkarias what are the cached variables of the VM loop that needs to be updated? I looked through the interpreter loop and can’t find any cached variables. Are you working on a lox like language yourself? What is your solution?

[RevengerWizard]

It's been a while and I've finally been able to tackle correctly the issue of reentrancy by basically using the approch Lua follows. It's not really about cached variables, which, to be honest, I don't even think are a problem in and of themselves. Rather, the problem is that re-entering requires recursively recalling the interpreter loop.

[HallofFamer]

@RevengerWizard are you saying that all it takes is to call the runInterpreter function recursively in the C API, without needing any changes to any cached variables? And how is the performance impact? I heard Wren devs mention that making VM reentrant will slow it down.

[RevengerWizard]

All it takes is calling the runInterpreter function recursively in the C API and correctly keeping track of CallFrames of the native C functions basically. I didn't fully test the performance impact, but since the approch is basically the same as Lua 5.0.3 (as implemented in ldo.c and ldo.h) it shouldn't slow down anything. It didn't even take that much effort to implement it. I think the problem with Wren is the way its C API was designed, rather than any potential performance penalty. Wren is more complicated because it involves interactions with fibers, so that could also be a factor.

You can check out the dev branch of Teascript (my small scripting language) to know more about this. I've chose to use a C API approch similar to Lua, as I think it's more elegant and safer to use than directly manipulating the language values.

[mcfriend99]

Can you point me to the exact place in the teascript source where this is implemented?? @RevengerWizard

[mcfriend99]

@RevengerWizard I think I solved my own problem myself. My implementation had a bug before where the stack gets mangled once another native function runs in the call and because of that it kept returning the wrong return value for function calls in those scenarios. I fixed it already.

[HallofFamer]

I also fixed it in my own CLox implementation. Other than what @RevengerWizard already mentioned to call the run function recursively, it is also important to change how OP_RETURN behaves so it can exit the inner interpreter loop correctly when inside a closure. It is not as complex as I thought.