search

JuliaParallel / DistributedArrays.jl

Distributed Arrays in Julia
Other
197 stars 35 forks source link

DArray roadmap #4

Open jakebolewski opened 9 years ago

jakebolewski commented 9 years ago

object identity and equality

Update 30 September 2016 by @andreasnoack: I've removed all the scalar math and special functions. With the . syntax + broadcast(x::DArray) = map(x) they should work.

jiahao commented 9 years ago

For a first pass we can skip most of the vectorized functions - i.e. all the specfuns and most of the scalar mathematical functions.

ViralBShah commented 9 years ago

Or implement them with macros and conquer much of this list. :-)

I love this list. I am sure @alanedelman loves it too.

ViralBShah commented 9 years ago

621 functions. 16 done. Only 605 to go. :-)

Cc: @amitmurthy @tanmaykm @alanedelman in case you guys haven't seen this.

jakebolewski commented 9 years ago

Some of these will be hard / impossible to do in parallel. My favorite is == if we want to support the same semantics as AbstractArrays now in base.

ViralBShah commented 9 years ago

Yes, of course!

What is the exact issue with == ? Perhaps should be a separate issue.

jiahao commented 9 years ago

@ViralBShah I agree == (with hash) should be their own issue. The challenge is how to hash DArrays in parallel so that they can test equal under == regardless of distribution. The only way we can guarantee that now for A==B is to redistribute all the data onto a single process and do the hash locally for A and B.

jakebolewski commented 9 years ago

@tanmaykm, @amitmurthy it would be great if you could check off functions as they are added. I believe anyone that is an owner of the repo can edit the check list.

alanedelman commented 9 years ago

don't we have svd and a few other scalapack functions now?

ViralBShah commented 9 years ago

We should probably separate out linear algebra related function checklist into scalapack.jl or elemental.jl, and have this issue track only DArrays.

alanedelman commented 9 years ago

but i'd like this to work with darrays :-)

On Thu, Apr 23, 2015 at 8:27 AM, Viral B. Shah notifications@github.com wrote:

We should probably separate out linear algebra related function checklist into scalapack.jl or elemental.jl, and have this issue track only DArrays.

— Reply to this email directly or view it on GitHub https://github.com/JuliaParallel/DistributedArrays.jl/issues/4#issuecomment-95568816 .

andreasnoack commented 9 years ago

It will, but the dependency order right now is that ScaLAPACK.jl depends on DistributedArrays.jl and, therefore, the DArray+ScaLAPACK functionality is in ScaLAPACK.jl.

andreasnoack commented 9 years ago

@alanedelman Just to avoid misunderstanding here. You can do

julia> using MPI

julia> man = MPIManager(np = 4);

julia> addprocs(man);

julia> using ScaLAPACK
Warning: Method definition defaultdistTuple{Int64,Int64} in module DistributedArrays at /Users/andreasnoack/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:147 overwritten in module ScaLAPACK at /Users/andreasnoack/.julia/v0.4/ScaLAPACK/src/ScaLAPACK.jl:13.
    From worker 4:  Warning: Method definition defaultdistTuple{Int64,Int64} in module DistributedArrays at /Users/andreasnoack/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:147 overwritten in module ScaLAPACK at /Users/andreasnoack/.julia/v0.4/ScaLAPACK/src/ScaLAPACK.jl:13.
    From worker 3:  Warning: Method definition defaultdistTuple{Int64,Int64} in module DistributedArrays at /Users/andreasnoack/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:147 overwritten in module ScaLAPACK at /Users/andreasnoack/.julia/v0.4/ScaLAPACK/src/ScaLAPACK.jl:13.
    From worker 2:  Warning: Method definition defaultdistTuple{Int64,Int64} in module DistributedArrays at /Users/andreasnoack/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:147 overwritten in module ScaLAPACK at /Users/andreasnoack/.julia/v0.4/ScaLAPACK/src/ScaLAPACK.jl:13.
    From worker 5:  Warning: Method definition defaultdistTuple{Int64,Int64} in module DistributedArrays at /Users/andreasnoack/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:147 overwritten in module ScaLAPACK at /Users/andreasnoack/.julia/v0.4/ScaLAPACK/src/ScaLAPACK.jl:13.

julia> @everywhere using DistributedArrays

julia> A = drandn((1000, 1000), man);

julia> B = convert(Array, A);

julia> @time valsA = svdvals(A);
elapsed time: 1.382340321 seconds (37 MB allocated, 1.53% gc time in 1 pauses with 0 full sweep)

julia> @time valsA = svdvals(A);
elapsed time: 0.323082764 seconds (1 MB allocated)

julia> @time valsB = svdvals(B);
elapsed time: 1.0682854 seconds (11 MB allocated, 0.48% gc time in 1 pauses with 0 full sweep)

julia> @time valsB = svdvals(B);
elapsed time: 0.998921006 seconds (8 MB allocated)```
The warnings will go away eventually.
alanedelman commented 7 years ago

I was kind of sad that now that we are about to have the "dot" (Pronounced broadcast but I don't like that name, I prefer pointwise or just point) why we don't have these functions after all these years.

Why can't we have all the operators and all the dot operators easily?

andreasnoack commented 7 years ago

We should have all the dot stuff now but it only works on Julia 0.6 and master. Somebody just needs to go through the list and check. Some of the items are also deprecated now.

andreasnoack commented 7 years ago

@alanedelman I just checked and all the dot stuff works now for DArrays. However, they are not independent functions anymore but just syntax so I've removed them from the list. The important detail is that broadcast works.

alanedelman commented 7 years ago

where are we on cumsum, cumprod, and accumulate ?

andreasnoack commented 7 years ago

Still some work to be done there.

alanedelman commented 6 years ago

would love to bring this issue back to life

montyvesselinov commented 5 years ago

me too

ViralBShah commented 4 years ago

@christopher-dG Just bringing this list to your attention. Maybe we should discuss this today?

alanedelman commented 4 years ago

+1