Closed xach closed 9 years ago
In many cases a custom task creator is sufficient,
(defun make-redirecting-task (fn &rest args)
(let ((out *my-stream*))
(lambda ()
(let ((*standard-output* out))
(apply fn args)))))
(submit-task channel (make-redirecting-task (lambda () (princ "foo"))))
Another way is to modify the existing bindings, which would require executing a task inside each worker. Curiously, lfarm has this functionality (needed for loading libs across servers), but not lparallel. The lparallel version is
(defun broadcast-task (function &rest args)
"Execute a task inside each worker and wait until all finish."
;; Eventually lparallel will depend on a thread util library that
;; includes semaphores. Until then, queues are stand-ins for
;; semaphores.
(let* ((*kernel* *kernel*)
(n (kernel-worker-count))
(channel (make-channel))
(from-workers (make-queue))
(to-workers (make-queue)))
(loop repeat n
do (submit-task channel (lambda ()
(push-queue t from-workers)
(pop-queue to-workers)
(apply function args))))
(loop repeat n do (pop-queue from-workers))
(loop repeat n do (push-queue t to-workers))
(loop repeat n do (receive-result channel))))
(let ((out *my-stream*))
(broadcast-task (lambda () (setf *standard-output* out))))
Making a new kernel will work too, of course. Including with-kernel
in the API is something I've considered several times, but I've always come out slightly against it. The goal is for the REPL to be useful without fuss; using with-kernel
means that a simple kill-tasks
from the REPL would not work, for instance. One would first need to track the temporary kernel and then write (let ((*kernel* ...)) (kill-tasks ...))
in the REPL.
I'm not exactly against with-kernel
-- I use it for testing -- but it carries complexities that are prone to cause confusion, especially for newcomers. It also takes seconds to write. Perhaps I could add it anyway, and just not use it in the examples in the documentation.
I like (and need) broadcast-task! It would be nice if it would be included in lparallel as well.
broadcast-task
was added to lparallel-2.7.5, which is now in quicklisp.
I considered with-kernel
again and decided against it (again) for the aforementioned reasons and others, including the fear of encouraging multiple with-kernel
s within an application that can defeat the purpose of having a thread pool.
@xach If broadcast-task
or make-redirecting-task
(above) aren't adequate, or if you have another suggestion, then let me know.
The latest version of lparallel has an unpublished utility lparallel.kernel-util:with-temp-kernel
with documentation explaining the dangers involved with its use. While it probably won't be included in the main API, it may be relied upon and considered stable.
I'm building some stuff with lparallel workers and I'd like to redirect the output. Can I modify the existing stream bindings in a kernel, or should I make a new kernel for the job?
If I have to make a new kernel, is there anything like with-kernel, so I don't have to worry about calling end-kernel?