Open dangirsh opened 5 years ago
Does any one tell me whether the macro "@distributed" work on iterators now? I have a very large data array (more than 900,000,000) after combinations, collect() function will exhaust all memory.
Yes, @distributed
still requires indexing — it'd indeed be good to document this requirement.
Note that distributing such a large array to your workers may swamp the computation time with communication overhead. You may want to consider "peeling off" one of your combinatorial elements, distribute that amongst your workers, and then have each of your workers compute the combinations of that one element with the remainder.
Thanks, mbauman. I will try your suggestions.
Hello Matt,
if I understand correctly, this indexing requirement is the reason behind eachrow
& eachcol
not working with @distributed
? Julia reports a syntax error saying 'invalid assignment location' in 1.2.
Yes, that's correct. Just as a breadcrumb, with https://github.com/JuliaLang/julia/pull/32310 we could possibly add indexing support.
Distributed
for
loops over iterators fail, seemingly because@distributed
expects the iterator to implementgetindex
. It makes sense that@distribute
needsgetindex
, but I don't see that in the docs. The possible solutions seem to be:@distributed
to mention this requirement.@distributed
to work for iterators.We could do JuliaLang/julia#1 until JuliaLang/julia#2 is finished.
Here's an example:
This is surprising, since the following non-parallel reduction works:
Adding a
collect
to the distributed version fixes it:Output:
600