search

JuliaParallel / DistributedArrays.jl

Distributed Arrays in Julia
Other
197 stars 35 forks source link

Update for julia-0.7 #156

Closed traktofon closed 6 years ago

traktofon commented 6 years ago

This change makes DistributedArrays work on julia 0.7, but compatibility with 0.6 is lost. All tests pass without deprecation warnings.

There are many minor syntax/library changes, and the following major changes:

  1. The broadcast implementation has been rewritten. The basic strategy is to rewrite the Broadcasted object into its equivalent for the localparts. This will probably only work if all involved arrays are distributed in a "compatible" way. More work is needed to make it fully general.
  2. localindexes renamed to localindices, reflecting the current convention in Julia.
  3. deepcopy(::DArray) added.
  4. importall Base was removed.

Note that there seems to be an obscure issue in Julia which one currently has to work around by calling using DistributedArrays on the master process before calling @everywhere using DistributedArrays.

Calls to reduce and mapreduce with the dims keyword (which replace the reducedim and mapreducedim functions) are currently slow and I guess some part is falling back to methods for AbstractArray. There is related code in src/darray.jl which is still getting called, but I haven't managed to figure out the call path from the Base method.

andreasnoack commented 6 years ago

Thanks. I might try to continue from here. CuArrays has a switch to turn off scalar indexing which is very useful for testing to make sure that we don't hit the generic fallbacks and I plan to add something similar here. It should make it easier to fix the mapreduce code.

andreasnoack commented 6 years ago

I think I've now managed to fix all the cases that hit the slow fallbacks and are covered by tests. The tests now set DistributedArrays.allowscalar(false) except for a few places and will therefore error if a generic fallback is used.