Problem when passing variable in map with DistributedArray

[itsdfish]

Hi all-

In the past, I was able to pass a variable with a distributedarray while using map(). Now I receive a perplexing error. Here is a minimum working example that produces the error in some cases, but not others:

using Distributed
addprocs(4)
@everywhere using DistributedArrays
@everywhere f1(x) = sum(x) .+ 2
@everywhere function f2(x,y)
                println("y + 1: ",y + 1)
                sum(x)
            end
@everywhere f3(x,y) = sum(x) .+ y 
@everywhere function f4(x,y)
                z = y
                println("z: ",z)
                sum(x) .+ z
            end
@everywhere f5(x,y) = y + 2
x = [rand(1000,1000) for i in 1:100]
dist = distribute(x) 
y = 2.0
#works 
map(x->f1(x),dist)
#prints y
map(x->f2(x,y),dist)
#works
map(x->f3(x,2.0),dist)
#fails
map(x->f3(x,y),dist)
#fails
map(x->f4(x,y),dist)
#fails
map(x->f5(x,y),dist)

As you can see, I can add 1 to y and print it, but I cannot add it to the distributed array, assign another variable z to the value of y and use it, or manipulate it and return it. It works when I hardcode a value into the function, which is suboptimal.

Here is the error message:

BoundsError: attempt to access ()
  at index [1]
getindex(::Tuple{}, ::Int64) at tuple.jl:24
find_darray(::Tuple{}) at mapreduce.jl:30
find_darray(::Base.Broadcast.Extruded{DArray{Array{Float64,2},1,Array{Array{Float64,2},1}},Tuple{Bool},Tuple{Int64}}, ::Tuple{}) at mapreduce.jl:33
find_darray(::Tuple{Base.Broadcast.Extruded{DArray{Array{Float64,2},1,Array{Array{Float64,2},1}},Tuple{Bool},Tuple{Int64}}}) at mapreduce.jl:30
find_darray(::Base.Broadcast.Broadcasted{Base.Broadcast.ArrayStyle{DArray},Tuple{Base.OneTo{Int64}},getfield(Main, Symbol("##27#28")),Tuple{Base.Broadcast.Extruded{DArray{Array{Float64,2},1,Array{Array{Float64,2},1}},Tuple{Bool},Tuple{Int64}}}}) at mapreduce.jl:29
similar(::Base.Broadcast.Broadcasted{Base.Broadcast.ArrayStyle{DArray},Tuple{Base.OneTo{Int64}},getfield(Main, Symbol("##27#28")),Tuple{Base.Broadcast.Extruded{DArray{Array{Float64,2},1,Array{Array{Float64,2},1}},Tuple{Bool},Tuple{Int64}}}}, ::Type{Float64}) at mapreduce.jl:24
broadcast(::getfield(Main, Symbol("##27#28")), ::DArray{Array{Float64,2},1,Array{Array{Float64,2},1}}) at broadcast.jl:759
map(::Function, ::DArray{Array{Float64,2},1,Array{Array{Float64,2},1}}) at mapreduce.jl:6
top-level scope at none:0
include_string(::Module, ::String, ::String) at loading.jl:1002
include_string(::Module, ::String, ::String, ::Int64) at eval.jl:30
(::getfield(Atom, Symbol("##110#115")){String,Int64,String})() at eval.jl:89
withpath(::getfield(Atom, Symbol("##110#115")){String,Int64,String}, ::Nothing) at utils.jl:30
withpath at eval.jl:46 [inlined]
#109 at eval.jl:88 [inlined]
hideprompt(::getfield(Atom, Symbol("##109#114")){String,Int64,String}) at repl.jl:76
macro expansion at eval.jl:87 [inlined]
(::getfield(Atom, Symbol("##108#113")){Dict{String,Any}})() at task.jl:85

I am using Ubuntu 16.04. Here is my Julia info:

 versioninfo()
Julia Version 0.7.0
Commit a4cb80f3ed (2018-08-08 06:46 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, haswell)
Environment:
  JULIA_NUM_THREADS = 4
  JULIA_CMDSTAN_HOME = /home/dfish/cmdstan-2.17.1

(v0.7) pkg> st
    Status `~/.julia/environments/v0.7/Project.toml`
  [c52e3926] Atom v0.7.3
  [336ed68f] CSV v0.3.0
  [593b3428] CmdStan v4.1.0
  [a8cc5b0e] Crayons v1.0.0
  [a93c6f00] DataFrames v0.13.0
  [aaf54ef3] DistributedArrays v0.5.0
  [31c24e10] Distributions v0.16.2
  [682c06a0] JSON v0.19.0
  [e5e0dc1b] Juno v0.5.2
  [5fb14364] OhMyREPL v0.3.0
  [429524aa] Optim v0.16.0
  [d96e819e] Parameters v0.9.2
  [58dd65bb] Plotly v0.1.1
  [f0f68f2c] PlotlyJS v0.10.2
  [91a5bcdd] Plots v0.19.1
  [295af30f] Revise v0.6.6
  [60ddc479] StatPlots v0.8.0
  [2913bbd2] StatsBase v0.25.0
  [104b5d7c] WebSockets v1.0.0

[andreasnoack]

Yes. There are still some issues with the broadcasting functionality in Julia 1.0. I'm working on fixing it.

[itsdfish]

I noticed there were some similarities with issue #159, but I wasn't sure to what extent this behavior is related. Hopefully it provides some useful information. Thank you for your hard work fixing the bugs.

[andreasnoack]

Should be fixed by https://github.com/JuliaParallel/DistributedArrays.jl/pull/161