ERROR: On worker 4: Local instance of remote reference not found

[mdhe1248]

I couldn't reproduce this error with small dataset or small maxshift. Does anyone have any advice on this issue?

ERROR: On worker 4: 
Local instance of remote reference not found 
 in channel_from_id at ./multi.jl:801 
 in init_loc_flds at ./sharedarray.jl:337  
 in deserialize at ./sharedarray.jl:372 
 in deserialize_datatype at ./serialize.jl:837 
 in handle_deserialize at ./serialize.jl:571   
 in deserialize at ./serialize.jl:896     
 in deserialize_datatype at ./serialize.jl:837  
 in handle_deserialize at ./serialize.jl:571  
 in deserialize at ./serialize.jl:541 
 in ntuple at ./tuple.jl:65  
 in handle_deserialize at ./serialize.jl:562 
 in deserialize_msg at ./multi.jl:120   
 in message_handler_loop at ./multi.jl:1317  
 in process_tcp_streams at ./multi.jl:1276  
 in #638 at ./event.jl:68  
 in #remotecall_fetch#626(::Array{Any,1}, ::Function, ::Function, ::Base.Worker, ::RemoteChannel{Channel{Any}}, ::Vararg{Any,N}) at ./multi.jl:1070 
 in remotecall_fetch(::Function, ::Base.Worker, ::RemoteChannel{Channel{Any}}, ::Vararg{Any,N}) at ./multi.jl:1062
 in #remotecall_fetch#629(::Array{Any,1}, ::Function, ::Function, ::Int64, ::RemoteChannel{Channel{Any}}, ::Vararg{Any,N}) at ./multi.jl:1080 
 in remotecall_fetch(::Function, ::Int64, ::RemoteChannel{Channel{Any}}, ::Vararg{Any,N}) at ./multi.jl:1080 
 in macro expansion at /home/donghoon/.julia/v0.5/BlockRegistrationScheduler/src/RegisterDriver.jl:85 [inlined] 
 in (::RegisterDriver.##5#14{JLD.JldFile,Dict{Symbol,Any},SharedArray{Bool,1},SharedArray{Bool,1},RegisterDriver.#getnextidx#12,RemoteChannel{Channel{Any}},Array{RegisterWorkerAperturesMismatch.AperturesMismatch,1},ImageMetadata.ImageMeta{Float16,4,AxisArrays.AxisArray{Float16,4,SubArray{Float16,4,SharedArray{Float16,4},Tuple{Colon,Colon,UnitRange{Int64},UnitRange{Int64}},false},Tuple{AxisArrays.Axis{:x,Ranges.LinSpace{Quantity{Float64, Dimensions:{𝐋}, Units:{μm}}}},AxisArrays.Axis{:l,Ranges.LinSpacce{Quantity{Float64, Dimensions:{𝐋}, Units:{μm}}}},AxisArrays.Axis{:l,Ranges.LinSpace{Quantity{Float64, Dimensions:{𝐋}, Units:{μm}}}},AxisArrays.Axis{:z,Ranges.LinSpace{Quantity{Float64, Dimensions:{𝐋}, Units:{μm}}}},AxisArrays.Axis{:time,Ranges.LinSpace{Quantity{Float64, Dimensions:{𝐓}, Units:{s}}}}}}},Array{Dict{Symbol,Any},1}})() at ./task.jl:360 

...and 2 other exceptions.

[timholy]

Not sure. The maxshift should not have anything to do with it. How did you create img?

[mdhe1248]

I have tried several parameters. The error occurred when grid size is (22, 22, 11) and maxshift is (35, 35, 3).

My image is median filtered img:

julia> img
Float16 ImageMeta with:
  data: 4-dimensional AxisArray{Float16,4,...} with axes:
    :x, Ranges.linspace(0.0 μm,645.6629999999999 μm,1120)
    :l, Ranges.linspace(0.0 μm,622.583 μm,1080)
    :z, Ranges.linspace(5.084745762711851 μm,300.0 μm,59)
    :time, Ranges.linspace(0.0 s,360.0 s,181)
And data, a 1120×1080×59×181 SubArray{Float16,4,SharedArray{Float16,4},Tuple{Colon,Colon,UnitRange{Int64},UnitRange{Int64}},false}
  properties:
    imagineheader: <suppressed>

Below is a trimmed script I run. Well, it may takes 30 minutes to finish this. Data is in flash.wustl.edu:/mnt/donghoon_030/20170412

wpids = addprocs(21)   # launch 3 worker processes

using Images, Unitful, FixedSizeArrays, JLD, MAT, AxisArrays
using BlockRegistration, BlockRegistrationScheduler
using RegisterWorkerAperturesMismatch

# Some physical units we'll need
const μm = u"μm"  # micrometers
const s  = u"s"   # seconds

fn = "/mnt/donghoon_030/20170412/exp2_20170412.imagine";
fn_pre = "/mnt/donghoon_030/20170412/exp2_pre.imagine"
img1 = load(fn_pre);
roi = (:,:,2:60, 1:size(img1, 4));
img = view(img1, roi...)  # you could alternatively select a subset of times
fixedidx = (nimages(img)+1) >> 1 # Select our "fixed" image
fixed0 = view(img, timeaxis(img)(fixedidx))

σ = 25μm
bias = 112; #High threshold.
gridsize = (20,20,3)  # or you could use round(Int, Float64[50μm/x for x in ps]) for one aperture each 50μm
mxshift = (35,35,3)

####Image Filtering
ps = pixelspacing(img)
sigmahp = Float64[σ/x for x in ps]
sigmalp = [3,3,1]  # lowpass filtering is not currently recommended
pp = PreprocessSNF(bias, sigmalp, sigmahp)
fixed = pp(fixed0)

#### Set up the grid of apertures for aligning the images.
knots = map(d->linspace(1,size(fixed,d),gridsize[d]), (1:ndims(fixed)...))
algorithm = AperturesMismatch[AperturesMismatch(fixed, knots, mxshift, pp; pid=wpids[i]) for i = 1:length(wpids)]
mon = monitor(algorithm, (:Es, :cs, :Qs, :mmis));  
basename = splitext(splitdir(fn)[2])[1]
fileout = string(basename, ".mm")

#### Run
@time driver(fileout, algorithm, img, mon)
rmprocs(wpids) 

[mdhe1248]

Hm, I tried to repeat the error and yes, I could repeat whenever I input big(?) parameters. For example, in the script above, I tried roi = (:,:,2:60, 1:21);, gridsize = (22,22,11);, mxshift = (35,35,3);. This worked fine. If roi = (:,:,2:60, 1:22); or larger in the time axis, I got the error.

[timholy]

It's unlikely that I'll have time to debug this before I get back. Sorry about that. (Talk preparations...)

[mdhe1248]

Thanks for checking this issue.

For now, I am taking a bit longer way: I run the mismatch step with smaller data sets (10stacks). I think I can combine outputs, such as 'mmis', in the warping step.

On Apr 23, 2017 9:09 PM, "Tim Holy" notifications@github.com wrote:

It's unlikely that I'll have time to debug this before I get back. Sorry about that. (Talk preparations...)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/HolyLab/BlockRegistrationScheduler/issues/36#issuecomment-296508712, or mute the thread https://github.com/notifications/unsubscribe-auth/AMvfFa8CNQSMzE0oZGfTycF9hIhJzrRQks5rzARvgaJpZM4NE6xO .

[mdhe1248]

For now, my solution is inefficient. However, it works anyway.. I used for loop. I registered only a few stacks, e.g.,10 stacks, at a time to the same fixed image.

[timholy]

Great to have a workaround. To me this smells like a julia bug, and one that could be quite difficult to track down. Before putting in a lot of effort, I suspect it would be worth seeing whether switching to julia 0.6 fixes it.

[timholy]

See https://github.com/JuliaLang/julia/issues/22139.