Large micromanager stack gives ReadOnlyMemoryError

[jowch]

Hello, I have some large micromanager tiff stacks (50000+ images) that I would like to load as a memory-mapped stack. I get a ReadOnlyMemoryError when I run the following code.

julia> stack = OMETIFF.load(File{format"OMETIFF"}(stack_path); inmemory=false)
ERROR: LoadError: ReadOnlyMemoryError()

Additionally, when I use FileIO directly, I get

julia> stack = load(stack_path; inmemory=false)
Error encountered while load File{DataFormat{:OMETIFF}, String}("path/to/MMStack_Default.ome.tif").

Fatal error:
ERROR: ReadOnlyMemoryError()
Stacktrace:
 [1] handle_error(e::ReadOnlyMemoryError, q::Base.PkgId, bt::Vector{Union{Ptr{Nothing}, Base.InterpreterIP}})
   @ FileIO ~/.julia/packages/FileIO/DNKwN/src/error_handling.jl:61
 [2] handle_exceptions(exceptions::Vector{Tuple{Any, Union{Base.PkgId, Module}, Vector{T} where T}}, action::String)
   @ FileIO ~/.julia/packages/FileIO/DNKwN/src/error_handling.jl:56
 [3] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Formatted; options::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:inmemory,), Tuple{Bool}}})
   @ FileIO ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:228
 [4] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:inmemory,), Tuple{Bool}}})
   @ FileIO ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:185
 [5] load(::String; options::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:inmemory,), Tuple{Bool}}})
   @ FileIO ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:113
 [6] top-level scope
   @ REPL[6]:1

I would provide an example file but it is quite large. Perhaps there is a good way to deal with this? I can help investigate this as well.

Any pointers would be much appreciated.

PS: This image stack was also from an aborted acquisition, but I saw an earlier closed issue that dealt with that so its probably not related (?).

[tlnagy]

This is a little difficult to diagnose from that error message. What does

julia> OMETIFF.dump_omexml(stack_path)

give you? Also, could you try opening the TIFF file with https://github.com/tlnagy/TiffImages.jl?

julia> TiffImages.load(stack_path; mmap=true)

Does that work?

[jowch]

I have attached the XML dump (44MB inflated) here: ome-xml.tar.gz

When I use TiffImages.load it only loads the first part of the whole stack. The whole stack is massive (100GB) so memory mapping will be key.

[tlnagy]

TiffImages should memory map (with the mmap=true flag) it properly if it's a BigTIFF. If it's lots of separate 4GB ome.tif files than it doesn't have logic to handle the splits since that's contained in the OME data. Then it's better to stick to OMETIFF.jl.

[tlnagy]

As for diagnosing the problem, the only thing I can suggest is to copy paste line by line from the OMETIFF.load function into your REPL starting here https://github.com/tlnagy/OMETIFF.jl/blob/6f3555e25637ac71ce3d7f39e810e16ddb3f86c6/src/loader.jl#L33 and let me know where it crashes for you and what the output is prior.

Thanks for helping me diagnose this. Hopefully we can get to the bottom of it.

[jowch]

I looks like its dying on the initial io open call before the OMETIFF function is run. The snippet below is from Base.io. Doesn't seem like this is an issue with OMETIFF.

function open(f::Function, args...; kwargs...)
    io = open(args...; kwargs...)
    try
        f(io)
    finally
        close(io)
    end
end

[tlnagy]

That's weird given that both TiffImages.jl and OMETIFF.dump_omexml are able to open the file?

[tlnagy]

Given that this appears to not be a OMETIFF.jl error, I'm going to go ahead and close this.