Promise callbacks should be deferred whenever possible

[ivan4th]

As of now, promise error callbacks are always invoked immediately and promise success callbacks are only deferred if *promise-finish-hook* is set, e.g.:

(setf blackbird:*promise-finish-hook*
      #'(lambda (fn)
          (as:with-delay () (funcall fn))))

That's quite unlike js promises, where both success and error callbacks are always executed only when control reaches the event loop.

The problem is that code that uses immediate callbacks often needs to be written in very different style from the code that uses deferred callbacks, and changing blackbird:*promise-finish-hook* may break some of the code depending on which style it is using.

Although immediate callback invocation is faster, it may cause some very weird code flow. E.g. you may have some code like

(defun talk-to-server (message)
  (bb:with-promise (resolve reject)
    (set-message-handler
     #'(lambda (message)
         (resolve message)
         (clear-message-handler)))
    ;; send-message returns immediately
    (send-mesage message)))

(defun dialogue ()
  (bb:alet ((response (talk-to-server message-1)))
    (talk-to-server (make-message-2-for-response response))))

Let's see what happens in dialogue. First talk-to-server call is made, which returns a promise which will be resolved sometime in future when a message arrives from the server. Until the message arrives, there's a message handler installed that will resolve the promise. Then, when the message is received, the promise is resolved and second talk-to-server call is made. It sets a new message handler (which may override the previous one, ok), initiates message sending and returns a promise. The problem is that the first (resolve message) call from the first (talk-to-server) call returns at this point, and (clear-message-handler) is called, killing the newly installed message handler. Such very strange "blast from the past" thing made me spend about 3 hours trying to debug a similar problem in my MQTT driver. Other, even more weird "unwanted reentrancy" pattens may occur, too.

Deferring error callbacks is also important because such callbacks may activate other long promise chains.

I'd suggest either making some bb-async ASDF system which can be added to either cl-async or blackbird, and/or using asdf-system-connections to set blackbird:*promise-finish-hook* to delaying function when cl-async is available.

Perhaps error callbacks should be updated to use blackbird:*promise-finish-hook*, too.

[ivan4th]

The example may seem to be a bit dumb, in fact it was a bit more involved with a list of handlers being cleared after every handler is run.

Also, it may be worth adding something like :immediate t for the cases when one knows what he/she is doing, like resolving the promise at the very end of some async callback which is guaranteed to be called asynchronously.

[ivan4th]

Perhaps bb-async could also contain some useful wrappers for cl-async functionality, such as delay-promise (promise-based replacement for as:delay), spawn-wait (spawn a process and wait for its termination), mkdtemp-promise and so on.

[orthecreedence]

Sorry it took me so long to respond, this has been a very busy month for me (outside of lisp stuff).

I'm fine with introducing a change in blackbird that defers callbacks whenever possible (including error callbacks). The first thing to change is to update *promise-finish-hook* to work for error handling, I think.

I think it would be difficult to force changing the hook depending on the package, but what if the default finish hook itself was able to detect whether it was in an event loop?

(lambda (finish-fn)
  (if (and (find-package :cl-async)
           (symbol-value (intern '*event-base* :cl-async)))
      (as:delay finish-fn)
      (funcall finish-fn)))

Thoughts?