cpos not defined

[rogerberm]

NRRD.jl crashes on line 235: A = Mmap.mmap(iodata, Array{Traw,length(szraw)}, szraw,cpos; grow=false)

Perhaps cpos = position(iodata) missing?

[timholy]

Illustrative example? (Would be nice to add it to the tests.) And which julia verison?

[rogerberm]

Sure.

The problem seems to happen when loading large 3D files. I have created an empty stack (all zeros) of size 500x500x500 in Fiji for testing. Part of the problem has to do with how Fiji generates headers, plus a bug, it seems.

The test:

using Images
stack = load("stack.nrrd");

This gives me

ERROR: KeyError: key "microns" not found
 in getindex(::Dict{String,Any}, ::SubString{String}) at ./dict.jl:688
 in collect(::Base.Generator{Array{SubString{String},1},NRRD.##41#49}) at ./array.jl:307
 in get_axes(::Dict{String,Any}, ::Int64) at /home/user/.julia/v0.5/NRRD/src/NRRD.jl:791
 in #load#4(::String, ::Symbol, ::Function, ::FileIO.Stream{FileIO.DataFormat{:NRRD},IOStream}, ::Type{T}) at /home/user/.julia/v0.5/NRRD/src/NRRD.jl:210
 in (::NRRD.#kw##load)(::Array{Any,1}, ::NRRD.#load, ::FileIO.Stream{FileIO.DataFormat{:NRRD},IOStream}, ::Type{T}) at ./<missing>:0
 in open(::NRRD.##2#3{String,Symbol,Tuple{}}, ::FileIO.File{FileIO.DataFormat{:NRRD}}, ::String) at ./iostream.jl:113
 in anonymous at ./<missing>:?
 in eval(::Module, ::Any) at ./boot.jl:234
 in #load#17(::Array{Any,1}, ::Function, ::FileIO.File{FileIO.DataFormat{:NRRD}}) at /home/user/.julia/v0.5/FileIO/src/loadsave.jl:87
 in load(::FileIO.File{FileIO.DataFormat{:NRRD}}) at /home/user/.julia/v0.5/FileIO/src/loadsave.jl:75
 in #load#13(::Array{Any,1}, ::Function, ::String) at /home/user/.julia/v0.5/FileIO/src/loadsave.jl:45
 in load(::String) at /home/user/.julia/v0.5/FileIO/src/loadsave.jl:45

Now, Fiji generates header space units "microns", which is not in unit_string_dict (NRRD.jl line 106). I fixed this by changing microns to um in the header:

$ sed 's/microns/um/g' stack.nrrd > stack2.nrrd

Now...

using Images
stack = load("stack2.nrrd");

...gives me

Error encountered while loading "/tmp/stack2.nrrd".
Fatal error:
ERROR: DimensionMismatch("src length 3 disagrees with isspace Bool[false,false,false]")
 in copy_space!(::Array{Any,1}, ::Array{Unitful.Units{(Unitful.Unit{:Meter}(-6,1//1),),Unitful.Dimensions{(Unitful.Dimension{:Length}(1//1),)}},1}, ::BitArray{1}) at /home/user/.julia/v0.5/NRRD/src/NRRD.jl:1062
 in get_axes(::Dict{String,Any}, ::Int64) at /home/user/.julia/v0.5/NRRD/src/NRRD.jl:793
 in #load#4(::String, ::Symbol, ::Function, ::FileIO.Stream{FileIO.DataFormat{:NRRD},IOStream}, ::Type{T}) at /home/rbermude/.julia/v0.5/NRRD/src/NRRD.jl:210
 in (::NRRD.#kw##load)(::Array{Any,1}, ::NRRD.#load, ::FileIO.Stream{FileIO.DataFormat{:NRRD},IOStream}, ::Type{T}) at ./<missing>:0
 in open(::NRRD.##2#3{String,Symbol,Tuple{}}, ::FileIO.File{FileIO.DataFormat{:NRRD}}, ::String) at ./iostream.jl:113
 in anonymous at ./<missing>:?
 in eval(::Module, ::Any) at ./boot.jl:234
 in #load#17(::Array{Any,1}, ::Function, ::FileIO.File{FileIO.DataFormat{:NRRD}}) at /home/user/.julia/v0.5/FileIO/src/loadsave.jl:87
 in load(::FileIO.File{FileIO.DataFormat{:NRRD}}) at /home/user/.julia/v0.5/FileIO/src/loadsave.jl:75
 in #load#13(::Array{Any,1}, ::Function, ::String) at /home/user/.julia/v0.5/FileIO/src/loadsave.jl:45
 in load(::String) at /home/user/.julia/v0.5/FileIO/src/loadsave.jl:45

This apparently happens because isspace is initialized to falses in line 704, but is not updated because the code does not enter the condition on the lines following. The condition looks for "space" in the header dictionary, but Fiji does not generate it.

So, I added the space tag

$ sed '3 a space: left-posterior-superior' stack2.nrrd > stack3.nrrd

and then tested again:

using Images
stack = load("stack3.nrrd");

Here is where the error I originally reported happens:

ERROR: UndefVarError: cpos not defined
 in #load#4(::String, ::Symbol, ::Function, ::FileIO.Stream{FileIO.DataFormat{:NRRD},IOStream}, ::Type{T}) at /home/user/.julia/v0.5/NRRD/src/NRRD.jl:235
 in (::NRRD.#kw##load)(::Array{Any,1}, ::NRRD.#load, ::FileIO.Stream{FileIO.DataFormat{:NRRD},IOStream}, ::Type{T}) at ./<missing>:0
 in open(::NRRD.##2#3{String,Symbol,Tuple{}}, ::FileIO.File{FileIO.DataFormat{:NRRD}}, ::String) at ./iostream.jl:113
 in anonymous at ./<missing>:?
 in eval(::Module, ::Any) at ./boot.jl:234
 in #load#17(::Array{Any,1}, ::Function, ::FileIO.File{FileIO.DataFormat{:NRRD}}) at /home/user/.julia/v0.5/FileIO/src/loadsave.jl:87
 in load(::FileIO.File{FileIO.DataFormat{:NRRD}}) at /home/user/.julia/v0.5/FileIO/src/loadsave.jl:75
 in #load#13(::Array{Any,1}, ::Function, ::String) at /home/user/.julia/v0.5/FileIO/src/loadsave.jl:45
 in load(::String) at /home/user/.julia/v0.5/FileIO/src/loadsave.jl:45

There is a reference to a variable cpos that is not within the scope. The other part of the code where there is a reference to another cpos is on lines 1117-1118, where the variable is initialized as cpos = position(iodata). I repeated the same in the offending code, namely inserted cpos = position(iodata) before line 235. This fixed the problem for me.

julia> using Images
julia> size(load("stack3.nrrd"))
(500,500,500)

I hope this information helps.

Julia and environment info:

julia> versioninfo()
Julia Version 0.5.1-pre+31
Commit 6a1e339 (2016-11-17 17:50 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, ivybridge)

Update

Just this is enough to replicate the original error:

using Images
save("/tmp/volume.nrrd", zeros(500,500,500));
v = load("/tmp/volume.nrrd");

Error encountered while loading "/tmp/volume.nrrd".
Fatal error:
ERROR: UndefVarError: cpos not defined
 in #load#4(::String, ::Symbol, ::Function, ::FileIO.Stream{FileIO.DataFormat{:NRRD},IOStream}, ::Type{T}) at /home/rbermude/.julia/v0.5/NRRD/src/NRRD.jl:235
 in (::NRRD.#kw##load)(::Array{Any,1}, ::NRRD.#load, ::FileIO.Stream{FileIO.DataFormat{:NRRD},IOStream}, ::Type{T}) at ./<missing>:0
 in open(::NRRD.##2#3{String,Symbol,Tuple{}}, ::FileIO.File{FileIO.DataFormat{:NRRD}}, ::String) at ./iostream.jl:113
 in anonymous at ./<missing>:?
 in eval(::Module, ::Any) at ./boot.jl:234
 in #load#17(::Array{Any,1}, ::Function, ::FileIO.File{FileIO.DataFormat{:NRRD}}) at /home/rbermude/.julia/v0.5/FileIO/src/loadsave.jl:87
 in load(::FileIO.File{FileIO.DataFormat{:NRRD}}) at /home/rbermude/.julia/v0.5/FileIO/src/loadsave.jl:75
 in #load#13(::Array{Any,1}, ::Function, ::String) at /home/rbermude/.julia/v0.5/FileIO/src/loadsave.jl:45
 in load(::String) at /home/rbermude/.julia/v0.5/FileIO/src/loadsave.jl:45

[timholy]

Thanks for a really useful bug report. See #11.

[timholy]

BTW, if you make an axis "temporal" in FIJI, it doesn't seem to encode that in the NRRD. Do you want to report that to Fiji, or should I?

[rogerberm]

Oh, I wasn't aware. Sure, please do. And thanks for the quick reply and patch!