lmj / lparallel

Parallelism for Common Lisp
http://lparallel.org
BSD 3-Clause "New" or "Revised" License
243 stars 29 forks source link

Cannot saturate cores #14

Closed sgkruk closed 10 years ago

sgkruk commented 10 years ago

First, thank you for lparallel. It is a very useful abstraction.

I do not understand the following and I'm hoping you can help. I have a job where over 99% of the time (according to the SBCL profiler in a version not using lparallel) is spent on calls to one function. This function is called thousands of times and each call is completely independent of every other call. It is therefore embarrassingly parallel. Yet, when I use lparallel to spread the work (I have tried with 2, 4 and 8 cores machines), I never saturate the cores, far from it. I use one channel for all tasks. Can you suggest where the bottleneck might be? Thanks for any help you can provide.

(defun filter-and-merge (M potentials) "Version with complete check. Parallel!" (setq version "Multiple threads. Tests split easy/hard.") (if (null potentials) M (let ((newM M) (channel (lparallel:make-channel)) (batchsize 1) (batchcount 0)) (do ((index 0 (+ index batchsize))) ((>= index (length potentials))) (incf batchcount)
(lparallel:submit-task channel #'wrap-slow-check-p M potentials index batchsize)) (for (batch 0 batchcount) (acctime (runtime count) (let ((results (lparallel:receive-result channel))) (dolist (r results) (when (cadr r) (push (car r) newM)))))) newM)))

lmj commented 10 years ago

@sgkruk, from what I can surmise, you create a separate task for each small computation and then sequentially duplicate all the results into a new list. I suspect pmap or pdotimes would be more suitable, though I don't have much context.

lmj commented 10 years ago

@sgkruk, I'm closing this because there doesn't seem to be an issue here. You are welcome to follow up by email.