async calls?

[bobzoller]

I've been attempting to add call_async, but I'm not having a lot of luck. I think it all boils down to push_proc not handling closures. Given that, I think the best I could do would to be to use some global JS object to hold the state and poll it from Crystal.

Has anyone (who's better at Crystal and C) given this more thought?

[jessedoyle]

Hey @bobzoller, thanks for usingduktape.cr!

I haven't really put much thought into how asynchronous calls from the Duktape VM to Crystal would work.

Duktape expects most of the communication between the interpreter and the host language to use its own internal stack. We can push Crystal methods onto the Duktape stack and call them. Unfortunately the fact that Crystal doesn't allow closures in C callback functions is a major drawback.

We may be able to use Crystal's own async mechanisms (spawn and Fiber.yield) to allow for async calls. This would take some experimentation. I'll look into this over the next few weeks to see if there are any easy solutions.

Would you be able to outline a bit more about your intended use-case? Ideally this may hint at implementation details in the future.

Finally, I'm open to any suggestions or PRs that outline a clean solution to asynchronous function calls.

[bobzoller]

Honestly it was an academic exercise to see if I could get React rendering on the (Crystal) server, and replicate something like react-rails.

I got a hello world React component rendering no problem, but of course the minute you introduce anything async (react-router, in my case), you're SOL without a way to wait on it.

At the end of the day, I've seen three ways of handling this in various other X to JS layers:

1. the javascript side calls a provided callback when it's done, and this directly calls Crystal code (eg your Proc support)
2. the javascript side changes some state when it's done, and Crystal is polling for that state change
3. the javascript side logs a special message, and Crystal is listening for that message (console output is often available as a stream)

Whichever way, then on the Crystal side you're using fibers or channels (or something higher level like crystal-futures) to wrap up the async bits to get a simple API.

As I said I'm new to Crystal and Duktape (and my C is beyond rusty), but I'll also try to dig in a little further and see what might be possible.

[jessedoyle]

Duktape 1.6.0 (which was just released) may have given us an option of async calls!

I'll have to update and give it a shot sometime soon!

See: https://github.com/svaarala/duktape/issues/834 and https://github.com/svaarala/duktape/pull/909

[jessedoyle]

I started spending some time investigating this and I realized that asynchronous calls are possible out of the box in this shard. We simply have to use Crystal's built-in concurrency model (spawn, channels, fibers).

The primary issue is that we are handcuffed - Crystal cannot directly pass a closure to a C function. This makes it difficult to work with Channel instances and communicate between fibers.

Although we may not be able to directly pass closure data to a Duktape function, we can pass arbitrary data through the stack as a raw pointer.

Therefore it's possible to decompose the closure data to a pointer, pass that pointer through the stack, then reconstitute the data back to its original form.

Here's an example I've been experimenting with:

require "duktape"

sbx = Duktape::Sandbox.new
channel = Channel(Int32).new

sbx.push_global_proc("closure", 2) do |ptr|
  env = Duktape::Sandbox.new(ptr)
  raw = env.require_pointer(0)  # argument 1
  timeout = env.require_int(1)  # argument 2 
  chan = raw.as(Channel(Int32)) # coerce back to a Channel(Int32)
  puts "closure(): called with #{chan.inspect}, #{timeout}"
  spawn do
    puts "closure(): sleeping for #{timeout} seconds"
    sleep timeout
    puts "closure(): sending channel data"
    chan.send(timeout)
  end
  env.call_success
end

3.times do
  sbx.push_global_object
  sbx << "closure"                    # property name
  sbx.push_pointer(channel.as(Void*)) # argument 1
  sbx << rand(10) + 1                 # argument 2
  sbx.call_prop(-4, 2)                # call property on global object with 2 arguments
end

3.times do
  puts "main(): before channel receive"
  val = channel.receive
  puts "main(): after channel receive - #{val}"
end

The code above produces output such as this:

closure(): called with #<Channel::Unbuffered(Int32):0x10a3cfb10>, 9
closure(): called with #<Channel::Unbuffered(Int32):0x10a3cfb10>, 1
closure(): called with #<Channel::Unbuffered(Int32):0x10a3cfb10>, 6
main(): before channel receive
closure(): sleeping for 9 seconds
closure(): sleeping for 1 seconds
closure(): sleeping for 6 seconds
closure(): sending channel data
main(): after channel receive - 1
main(): before channel receive
closure(): sending channel data
main(): after channel receive - 6
main(): before channel receive
closure(): sending channel data
main(): after channel receive - 9

@bobzoller - I know it's been some time since any headway has been made on this issue, but I think this is a solid approach that can be implemented out of the box.

I'm also going to reference this issue as I've realized that it is possible to pass closure data to a custom Duktape function.

I hope this helps!

[jessedoyle]

For now I'm going to close this issue as I believe the approach above should allow async calls.

Feel free to comment on this issue and I can open it again.