lmj / lparallel

Parallelism for Common Lisp
http://lparallel.org
BSD 3-Clause "New" or "Revised" License
243 stars 29 forks source link

Redirecting output of an existing kernel? #16

Closed xach closed 9 years ago

xach commented 9 years ago

I'm building some stuff with lparallel workers and I'd like to redirect the output. Can I modify the existing stream bindings in a kernel, or should I make a new kernel for the job?

If I have to make a new kernel, is there anything like with-kernel, so I don't have to worry about calling end-kernel?

lmj commented 9 years ago

In many cases a custom task creator is sufficient,

(defun make-redirecting-task (fn &rest args)
  (let ((out *my-stream*))
    (lambda ()
      (let ((*standard-output* out))
        (apply fn args)))))

(submit-task channel (make-redirecting-task (lambda () (princ "foo"))))

Another way is to modify the existing bindings, which would require executing a task inside each worker. Curiously, lfarm has this functionality (needed for loading libs across servers), but not lparallel. The lparallel version is

(defun broadcast-task (function &rest args)
  "Execute a task inside each worker and wait until all finish."
  ;; Eventually lparallel will depend on a thread util library that
  ;; includes semaphores. Until then, queues are stand-ins for
  ;; semaphores.
  (let* ((*kernel* *kernel*)
         (n (kernel-worker-count))
         (channel (make-channel))
         (from-workers (make-queue))
         (to-workers (make-queue)))
    (loop repeat n
          do (submit-task channel (lambda ()
                                    (push-queue t from-workers)
                                    (pop-queue to-workers)
                                    (apply function args))))
    (loop repeat n do (pop-queue from-workers))
    (loop repeat n do (push-queue t to-workers))
    (loop repeat n do (receive-result channel))))

(let ((out *my-stream*))
  (broadcast-task (lambda () (setf *standard-output* out))))

Making a new kernel will work too, of course. Including with-kernel in the API is something I've considered several times, but I've always come out slightly against it. The goal is for the REPL to be useful without fuss; using with-kernel means that a simple kill-tasks from the REPL would not work, for instance. One would first need to track the temporary kernel and then write (let ((*kernel* ...)) (kill-tasks ...)) in the REPL.

I'm not exactly against with-kernel -- I use it for testing -- but it carries complexities that are prone to cause confusion, especially for newcomers. It also takes seconds to write. Perhaps I could add it anyway, and just not use it in the examples in the documentation.

nneuss commented 9 years ago

I like (and need) broadcast-task! It would be nice if it would be included in lparallel as well.

lmj commented 9 years ago

broadcast-task was added to lparallel-2.7.5, which is now in quicklisp.

I considered with-kernel again and decided against it (again) for the aforementioned reasons and others, including the fear of encouraging multiple with-kernels within an application that can defeat the purpose of having a thread pool.

@xach If broadcast-task or make-redirecting-task (above) aren't adequate, or if you have another suggestion, then let me know.

lmj commented 9 years ago

The latest version of lparallel has an unpublished utility lparallel.kernel-util:with-temp-kernel with documentation explaining the dangers involved with its use. While it probably won't be included in the main API, it may be relied upon and considered stable.