Setting remote values

[r-barnes]

Based on the following tests:

@everywhere using DistributedArrays
ras    = [@spawnat p rand(30,30) for p in workers()[1:4]]
ras    = reshape(ras,(2,2))
D      = DArray(ras)
D[1,1] = 5
m1     = @spawnat 2 D[:L][2,2] = 5 #Works
m2     = @spawnat 2 D[2,2] = 5     #Doesn't work
fetch(m1)
fetch(m2)

It looks as though DArrays offers view-only access to non-local portions of the array and write access only to local portions. Is that correct?

Any thoughts on how to set remote values?

[vchuravy]

Yes see #155 for the reasoning behind it.

I = (2,2)
chidx = locate(D, I...)
idxs = d.indices[chidx...]
localidx = ntuple(i -> (I[i] - first(idxs[i]) + 1), ndims(d))
pid = d.pids[chidx...]

remote_call(pid, D, localidx, data) do D, I, data
     localpart(D)[localidx...] = data
end

Which if your data is not a scalar should be reasonable efficient. This is based of https://github.com/JuliaParallel/DistributedArrays.jl/blob/e6915d28b59f564043597985bd5af1896f3e380b/src/darray.jl#L620 and might need some adjustment to deal with slices. https://github.com/JuliaParallel/DistributedArrays.jl/blob/e6915d28b59f564043597985bd5af1896f3e380b/src/darray.jl#L347 does that and so we should probably factor those parts out.

[r-barnes]

Got it. I'm planning to use DArray with SparseArrays to form a large binary matrix representing the frontier of a breadth-first search, so I can send the index updates en masse after each round provided I know where to send them. I can figure that out based on the snippet you provided.

[vchuravy]

We now have DistributedArrays.allowscalar(false) so I think we could add setindex! in general and optimise the batch case.

[r-barnes]

Per my other post, I'm worried about communication delays, so I'm thinking about bulk exchange of halos now.