search

JuliaLang / julia

The Julia Programming Language
https://julialang.org/
MIT License
45.43k stars 5.45k forks source link

add comprehension support to `@threads` #209

Open StefanKarpinski opened 12 years ago

StefanKarpinski commented 12 years ago

For example:

@parallel [ f(i) | i=1:n ]
ajdecon commented 12 years ago

Is this particularly different from pmap(f, {i|i=1:10} )?

I.e., would it make sense to implement this kind of comprehension as just some syntactic sugar on pmap, or are you envisioning more to it than that?

JeffBezanson commented 12 years ago

The syntactic sugar part could be @parallel [ f(i) | i = 1:n ].

But, we want to handle multiple dimensions, and it would be nice for it to return a distributed array. It would also be nice for each processor to compute points evenly spaced over the whole range, since this usually gives better load balancing. Then you might want to be able to move those results into a simpler distribution.

JeffBezanson commented 11 years ago

Now implemented on the jb/multi branch!

StefanKarpinski commented 11 years ago

Sweet. Is this an indication that the new Darray model is more productive?

JeffBezanson commented 11 years ago

Yes, a bit, but the only real difference in the interface is that the core operation is constructing a piece of an array from its indexes. The old DArray constructor could have been changed to work that way, but many other changes are needed too (distributing in all dimensions, and doing most operations lazily).

I'm also not sure what to do about array references inside parallel comprehensions. What you want is for A[i,j] to be transformed to A_chunk[i-ioffs,j-joffs] where A_chunk is the part of A that each processor needs.

JeffBezanson commented 11 years ago

The basic thing is now on master, but it does not handle array references yet. Doing that rewrite may be too magical, but we should probably do it.

amitmurthy commented 10 years ago

@JeffBezanson , seems like you had already implemented this - which I have moved under @darray in https://github.com/JuliaLang/julia/pull/5512 .

What do you mean by "does not handle array references yet" ?

StefanKarpinski commented 4 years ago

This would be spelled @threads [ f(i) for i in 1:n ] these days. Not sure if we need to keep an issue open for it, but I don't want to deprive someone of the fun of closing a three-digit issue when they implement it 😁

tkf commented 4 years ago

FYI,

using Transducers
tcollect(2x for x in 1:10)

already does it in parallel. Filtering and flattening are also supported.