Is there an equivalent of "sleep" in cl-async?

[chrisk414]

Hi, again! The original reason I'm interested in cl-async is to look for lightweight solution to replace heavy os "thread" And "sleep" is a must have. I think I can use nested "delays" to emulate this behavior but it's probably not a good idea because it can get quite deep pretty quickly. How should go about this? Thanks.

[orthecreedence]

To answer your question, no, there is no sleep. delay is the function you want to use.

This is the problem with async code...it must be nested, sometimes many layers deep. I wrote up a post on the cl-async site about this, but there's not much of a solution to the nesting problem.

One way to solve it is using futures, bundled with cl-async under the cl-async-future package:

(defun sleep-async (seconds)
  (let ((future (make-future)))
    (delay (lambda () (finish future)) :time seconds)
    future))

(wait-for (sleep-async 3)
  (format t "called 3 sec after running~%")
  (wait-for (sleep-async 2)
    (format t "called 5 sec after running~%")
    (wait-for ...)))

If you don't like the syntax of CPS (continuation passing style, the style cl-async uses you pass callbacks around instead of returning values), you really may want to look into the futures page of the documentation and learn some of the syntax helper macros that make things more natural. You're still going to be nesting, but it will look a lot more like real lisp code instead of a thousand (lambda ...) blocks everywhere.

Another solution is to use cl-cont. Another lisper has done great work building lightweight threading on top of cl-cont. It's hard, however, to mix non-cl-cont code with code that uses it, but that doesn't mean it's not worth seeing if it can be used to solve your specific problem.

[chrisk414]

Thanks for the pointer. I'll study them at first chance. Have a nice day~

[chrisk414]

Hi, I read your post and you are using (sleeper) in non-nested fashion. Code snippet is not complete so I'm not sure how it's done.

(defun my-app ()
  (format t "Now sleeping for 2s.~%")
  (sleeper)  ; this now blocks
  ;; execution resumes on the same stack
  (format t "Done sleeping!~%"))

[chrisk414]

Sorry about my last post. I read your article little too quick.^^ I guess there is no easy solution but to resort to CPS. I'm thinking about defining macro to make it easier. On the surface, it's squence of functions but the macro will convert it to CPS internally. Hmm...

[orthecreedence]

@chrisk414, you are going to inadvertently reinvent cl-cont, which will be no small feat. I suggest you study up a little bit on continuations (available in scheme, but not common lisp), and then take a look at cl-cont (macro-based continuations for common lisp). Once you have an ok understanding of cl-cont (it took me quite a while to wrap my head around it), check out the green threads library I pointed you towards.

[chrisk414]

Perhaps I'm heading down to the same path.. I took a quick look at green-thread but doesn't seem to show how "sleep" is supported. Am I missing something? ^^

[orthecreedence]

Take a look at this post. The author of green-threads posted it to show me how to integrate cl-async and green-threads. It's a bit old, so code on both green-threads and cl-async may be deprecated here and there, but reading over it should give you an idea of how the green-threads library works and how to integrate it.

[chrisk414]

Holy Matrimony!!! It works beautifully!!! It will take time to soak in how everything works but I'm so jolly that it works!! Once again, I really appreciate for your help. Have a great day!

[orthecreedence]

Glad to hear it, let me know if you have any other questions =].

[chrisk414]

Opps, I celebrated little too quick. "sleep" like call, (green-threads:wait-on (f-delay :time time))) require to be called within green-threads:with-green-thread macro, otherwise it won't work. This means that I cannot call "sleep" from a function. It severly reduces the usability. I'll investigate little further to see what can be done.

Here is example code I'm using that illustrates the problem. I'm calling "do-something-with-delay" and trying to put "sleep" code there. I'm first trying to understand what's going on but "continuation" stuff is not so easy to understand. :(

(defun f-delay (&key time)
  (let ((delay-future (green-threads:make-future)))
    (funcall #'cl-async:delay
         #'(lambda ()
         (green-threads:complete-future
          delay-future))
         :time time)
    delay-future))

(defun do-something-with-delay (time)
  (green-threads:wait-on (f-delay :time time))
  (print "doing something...")
 )

(defun test ()
  (cl-async:start-event-loop
   (lambda ()
     ;; Delay thread
     (green-threads:with-green-thread
       (format t "Starting 3 second delay.~%")
       (do-something-with-delay 3)
       ;;(green-threads:wait-on (f-delay :time 3.0))
       (format t "Finished 3 second delay.~%")
       )
     )
   ))

[orthecreedence]

Yep, you hit one of the limitations of cl-cont: all code that depends on cl-cont must be wrapped in cl-cont macros (like I said, it's hard to mix cl-cont code with non-cl-cont code).

One way to solve it is to replace

(defun do-something-with-delay ...)

with

(defun/cc do-something-with-delay ...)

(defun/cc is part of the cl-cont package). This creates an abstraction for you. Basically, all functions that have any async code in them that you want to not use CPS with must be wrapped in cl-cont macros: either with-green-thread, defun/cc, without-call/cc.

Without coroutine support, the only way to get rid of CPS is with code-walking macros like cl-cont, and the only way to get those to work uniformly is to wrap all of your code in macros that recognize what you're trying to do.

It's a pain in the ass, but welcome to async programming =].

[chrisk414]

ok, I understand the limitation. Depends on the situation, I use the version that's more convenient(less pain the the ass.. kk). Thanks for the explanation. Cheers!

[chrisk414]

@orthecreedence, I need little advice before I start endeavoring to convert my codes using cl-async. Looking at the expanded macros generated by cl-cont generates quite a bit of overheads for seemingly simple code. I'm little concerned about it's effect on performance. Have you had some knowledge in this area?

(cl-cont:with-call/cc (lambda nil
            (print 1)
            (print 2)
            (print 3)           
            ))))

expands to

(funcall #'values
         (cl-cont::make-funcallable
           (lambda (#:g7379)
             (declare (ignorable #:g7379))
             (funcall (lambda (&optional #:g7392 &rest #:g7393)
                        (declare (ignorable #:g7392))
                        (declare (ignore #:g7393))
                        (funcall (lambda
                                  (&optional #:g7394 &rest #:g7395)
                                  (declare (ignorable #:g7394))
                                  (declare (ignore #:g7395))
                                  (cl-cont::funcall/cc
                                   #:g7392
                                   (lambda
                                    (&optional #:g7380 &rest #:g7381)
                                    (declare (ignore #:g7380 #:g7381))
                                    (funcall
                                     (lambda
                                      (&optional #:g7388 &rest #:g7389)
                                      (declare (ignorable #:g7388))
                                      (declare (ignore #:g7389))
                                      (funcall
                                       (lambda
                                        (&optional
                                         #:g7390
                                         &rest
                                         #:g7391)
                                        (declare (ignorable #:g7390))
                                        (declare (ignore #:g7391))
                                        (cl-cont::funcall/cc
                                         #:g7388
                                         (lambda
                                          (&optional
                                           #:g7382
                                           &rest
                                           #:g7383)
                                          (declare
                                           (ignore #:g7382 #:g7383))
                                          (funcall
                                           (lambda
                                            (&optional
                                             #:g7384
                                             &rest
                                             #:g7385)
                                            (declare
                                             (ignorable #:g7384))
                                            (declare (ignore #:g7385))
                                            (funcall
                                             (lambda
                                              (&optional
                                               #:g7386
                                               &rest
                                               #:g7387)
                                              (declare
                                               (ignorable #:g7386))
                                              (declare
                                               (ignore #:g7387))
                                              (cl-cont::funcall/cc
                                               #:g7384
                                               #:g7379
                                               #:g7386))
                                             3))
                                           #'print))
                                         #:g7390))
                                       2))
                                     #'print))
                                   #:g7394))
                                 1))
                      #'print))))

[orthecreedence]

So what's happening is that

(print 1)
(print 2)
(print 3)

is being turned into CPS under the hood

(lambda ()
  (print 1)
  (lambda ()
    (print 2)
    (lambda ()
      (print 3))))

That's not an entirely accurate representation, but hopefully you get the idea. cl-cont does this by "walking the code" and re-arranging anything inside the with-call/cc to be CPS. So it doesn't really provide continuations in the sense you get them in scheme, it's really just rearranging your code to make it look like you have continuations...which is why you need to wrap anything that uses one of these "continuations" in a without-call/cc or defun/cc wrapper...so that code understands that it's dealing with rearranged code.

A lot of people find this approach messy. I can't really disagree, and have shied away from using cl-cont too heavily, but it does what it does fairly well. Keep in mind that debugging this code can be a nightmare. What used to be a simple function call is now buried in 20 layers of anonymous functions.

As far as performance goes, I urge you to do your own testing. Use the time function and run a few basic looping tests. My guess is that while cl-cont is going to be a little slower, it's not going to be materially slower. Lisp is surprisingly fast at creating and calling functions on the fly, at least in my experience, and most of the heavy processing is done via macros so by the time your code ever runs, the heavy lifting is over.

My suggestion is to use cl-cont for places where you're going to be heavily nesting your async code. If you really only need one or two layers, maybe having a lambda or two laying around can't hurt, and you can always use futures to mitigate some of the async ugliness.

[orthecreedence]

One more thing, green-threads just converted to use cl-async's futures implementation, so when you call green-thread's wait-on, you can pass it a cl-async future and it will pause the thread until that future finishes.

[chrisk414]

Thanks. Good advices. The more I look at cl-cont, the more I get turned off by it. I'll focus on cl-async CPS. Anyway, things are much clearer now and I begin to appreciate cl-async CPS little more.

Next, I need to think about how to control individual "delay", such as pause, preemtive kill within the nested environment. So if I preemtively kill a delay (meaning, a immediate kill without executing the callback to delay), all other delays below the current also get killed. Is this possible? ^^

[orthecreedence]

Maybe if you told me what your ultimate goal was I could help you, but right now I'm not really understanding what you're asking.

[chrisk414]

oh. sorry, to make matters simple, how do I kill(or return from) a "delay" immediately without executing the callback?

[orthecreedence]

Hmm, that's something I never thought of. Right now, it wouldn't be possible through cl-async's current API, but I could update delay to return the event object it uses to track the delay so it could be manually freed/cancelled.

Another option is to do something like this:

(defun cancelable-delay (delay-fn &key time event-cb)
  "Create a delay, and return a function that cancels execution of the delay
   upon calling."
  (let* ((cancel nil)
         (cancel-fn (lambda () (setf cancel t))))
    (as:delay
      (lambda ()
        (unless cancel
          (funcall delay-fn)))
      :time time
      :event-cb event-cb)
    cancel-fn))

;; example usage
(let ((cancel-fn (cancelable-delay
                   (lambda ()
                     (format t "hi~%"))
                   :time 2)))
  (when (should-cancel)
    (funcall cancel-fn)))

So cancelable-delay returns a function that when called, cancels execution of the delay function. This should achieve what you want without adding much overhead at all.

[chrisk414]

Great!!! I'll give it a try.. controlling delays are necessary for AI application as delays often needs be canceled or paused during simulations. While we are at it. It is possible to "pause" the delay timer until "resume" is requested? When "resumed", delays picks up the remaining timer instead of begining from the fresh. I believe cancelable and pausable delays can add to our powerful async arsenals. Thanks a lot for your help.

[orthecreedence]

Once again, this is not programmed into cl-async, but I believe could be solved outside of it. I don't have time to get into an in-depth example right now, but consider the cancelable-delay function. You could expand it so that when it calls as:delay, it marks the start time it used by making note of get-internal-real-time (into a var called start-time, let's say). Instead of just returning a cancel function, it could also return a pause function as a second value and a resume function as a third value. The pause function essentially cancels the delay and using get-internal-real-time, calculates how much time remains on the original delay: (- delay-time (- pause-time start-time)). When the resume function is called, it creates a new as:delay call, but with the remaining delay time that the pause function recorded.

Remember to divide time differences you get from get-internal-real-time by internal-time-units-per-second to give you seconds, which can be directly subtracted from the delay time.

[chrisk414]

Thanks for the suggestion. I took a look at the first example, cancelable-delay. I guess I wanted preemtive version canceling of delay, instead of waiting for the time-out. Every delay need be cancelable, pausable.. and I was wandering if it can be supported at the lower level. This is probably has to go little deeper than writting a wrapper, perhaps dealing with libevent. Is this possible? Hmm... Thanks.

[orthecreedence]

You can get cancelable delays at the lower level, with some mods to cl-async. Pause/resume, however, will have to be done in a manner similar to what I suggested...as far as I know libevent doesn't support pausing/disabling an event. You have to free it before it fires, and then create a new even with the time difference. I may be wrong here as well, but from what I know about libevent's API, all you get is cancel.

Please add another issue for supporting "cancel" with delays and I'll write something up when I can this week!