enhanced thread support

[milosn]

At the moment if you do something like:

... %context of another thread, not the one where loop is running% (as:delay (lambda () (as:write-socket-data my-sock my-data))) ....

you get:

debugger invoked on a SIMPLE-ERROR in thread THREAD "Anonymous thread" RUNNING {1007C468C3} : Event loop not running. Start with function start-event-loop.

Is it possible to tweak as:delay to somehow find the loop thread and register function call in that thread? Or is there another way to make this kind of thing work? Its quite useful I think.

Regards Milos

[orthecreedence]

This is an area I really haven't explored well. Right now there are a number of state variables required to track callbacks and data, making it so you'd have to copy each one out into your new thread.

What I'd like to do, now that you bring it up is to create an event-base class which holds all of its state in one object. Then running on another thread would be as simple as:

(defparameter *global-event-base* nil)

(defun thread-me ()
  (let ((cl-async-util:*event-base* *global-event-base*))
    ...))

(defun start ()
  (as:start-event-loop
    (lambda ()
      (setf *global-event-base* cl-async-util:*event-base*)
      (bt:make-thread #'thread-me)
      ...)))

So right you, if you want this to work, you'll have to copy the following vars from the main thread into the new one (like we did above with *event-base*):

• cl-async-util:*event-base*
• cl-async-util:*event-base-id*
• cl-async-util:*fn-registry*
• cl-async-util:*data-registry*
• cl-async-util:*event-loop-end-functions*

*dns-base* and *dns-ref-count* might be best if they are left to their own thread (not copied) since there's not a lot of benefit to having one dns base for two threads vs two dns bases for two threads (and the dns bases are created/destroyed so often by cl-async, that you'd have to do a lot of work copying them from the main thread).

So making all those variables part of a unified event-base class (and adding a (get-event-base) function) would make threading easier. One thing to note is that we'd probably have to have a switch in the event base object to allow threading, and if it's enabled anything adding/reading events to/from it will have to lock. Libevent does some of this already internally (if you tell it to), but we'd need to do our own locking for our own objects.

Thanks for the ideas!

[milosn]

Ok, thanks for prompt reply. Ill play around with your suggestion and lets see how far i get. But yes, I assumed some level of locking will be needed.

On the reasons why i need this ... Ive done multiple projects in python over past few years using Twisted async framework. Basically sooner or later you run into situation where you do need to use threads (blocking DB driver is one example), most of the time you can return your result from the thread and sched response writing in the main loop thread, but sometimes its necessary to be able to manipulate main event loop directly from threads other then the one where loop is running. Playing nice with threads its absolutely necessary :-) IMO.

Just as an example of different interfaces they provide (python is much different to lisp, but IMO its good reference as their threading interface evolved over the years from the need in real world projects)

http://twistedmatrix.com/documents/current/core/howto/threading.html

Regards Milos

[orthecreedence]

Thanks for the link and the use cases. It will help me structure things better for threading support. It may be a bit longer until I get it going since there are some outstanding issues (platform-specific bugs, mainly) I'd like to fix before adding any more features. I'll update this issue as I make progress.

Also, although it's not a replacement for threading support in cl-async, you might want to check out pretend-event-loop. I wrote it a while back before touching real async stuff. It has passive/worker thread pools that you can schedule tasks to (ie, a blocking DB driver). It's not as full-featured as I'd like, but it could easily be used in conjunction with cl-async to use blocking drivers until they're converted to async.

[orthecreedence]

Marking this as closed for now, just did a large number of threading updates: http://orthecreedence.github.io/cl-async/threading

Keep in mind threading support is very, very new.