search

BioJulia / Libz.jl

Fast, flexible zlib bindings.
Other
27 stars 17 forks source link

FileIO support #68

Open karlwessel opened 3 years ago

karlwessel commented 3 years ago

Expected Behavior

Since GZipped files are registered in FileIO to be opened with this library I would expect 

using FileIO
load("somezippedfile.gz")

or at least the streaming version

loadstreaming("somezippedfile.gz")

to work by giving me some unzipped file handle or a stream.

Current Behavior

However both give me an error. For load(...) it is 

Error encountered while load File{DataFormat{:GZIP},String}("work/tiptilt-auxiliaries/notebooks/shorttest.jl.gz").

Fatal error:
ERROR: UndefVarError: load not defined
action(::Symbol, ::Array{Union{Base.PkgId, Module},1}, ::Formatted; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:202
action at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:192 [inlined]
action(::Symbol, ::Array{Union{Base.PkgId, Module},1}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:181
action at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:181 [inlined]
load(::String; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:113
load(::String) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:110
top-level scope at REPL[5]:1
eval(::Module, ::Any) at ./boot.jl:331
eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:134
repl_backend_loop(::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:195
start_repl_backend(::REPL.REPLBackend, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:180
run_repl(::REPL.AbstractREPL, ::Any; backend_on_current_task::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:292
run_repl(::REPL.AbstractREPL, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288
(::Base.var"#807#809"{Bool,Bool,Bool,Bool})(::Module) at ./client.jl:399
#invokelatest#1 at ./essentials.jl:710 [inlined]
invokelatest at ./essentials.jl:709 [inlined]
run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:383
exec_options(::Base.JLOptions) at ./client.jl:313
_start() at ./client.jl:506
Stacktrace:
 [1] handle_error(::UndefVarError, ::Base.PkgId, ::Array{Union{Ptr{Nothing}, Base.InterpreterIP},1}) at /home/karl/.julia/packages/FileIO/BUPwf/src/error_handling.jl:61
 [2] handle_exceptions(::Array{Tuple{Any,Union{Base.PkgId, Module},Array{T,1} where T},1}, ::String) at /home/karl/.julia/packages/FileIO/BUPwf/src/error_handling.jl:56
 [3] action(::Symbol, ::Array{Union{Base.PkgId, Module},1}, ::Formatted; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:221
 [4] action at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:192 [inlined]
 [5] action(::Symbol, ::Array{Union{Base.PkgId, Module},1}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:181
 [6] action at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:181 [inlined]
 [7] load(::String; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:113
 [8] load(::String) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:110
 [9] top-level scope at REPL[5]:1

for loadstreaming(...) it is

Error encountered while loadstreaming File{DataFormat{:GZIP},String}("work/tiptilt-auxiliaries/notebooks/shorttest.jl.gz").

Fatal error:
ERROR: UndefVarError: loadstreaming not defined
action(::Symbol, ::Array{Union{Base.PkgId, Module},1}, ::Formatted; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:202
action at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:192 [inlined]
action(::Symbol, ::Array{Union{Base.PkgId, Module},1}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:181
action at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:181 [inlined]
loadstreaming(::String; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:113
loadstreaming(::String) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:110
top-level scope at REPL[6]:1
eval(::Module, ::Any) at ./boot.jl:331
eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:134
repl_backend_loop(::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:195
start_repl_backend(::REPL.REPLBackend, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:180
run_repl(::REPL.AbstractREPL, ::Any; backend_on_current_task::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:292
run_repl(::REPL.AbstractREPL, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288
(::Base.var"#807#809"{Bool,Bool,Bool,Bool})(::Module) at ./client.jl:399
#invokelatest#1 at ./essentials.jl:710 [inlined]
invokelatest at ./essentials.jl:709 [inlined]
run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:383
exec_options(::Base.JLOptions) at ./client.jl:313
_start() at ./client.jl:506
Stacktrace:
 [1] handle_error(::UndefVarError, ::Base.PkgId, ::Array{Union{Ptr{Nothing}, Base.InterpreterIP},1}) at /home/karl/.julia/packages/FileIO/BUPwf/src/error_handling.jl:61
 [2] handle_exceptions(::Array{Tuple{Any,Union{Base.PkgId, Module},Array{T,1} where T},1}, ::String) at /home/karl/.julia/packages/FileIO/BUPwf/src/error_handling.jl:56
 [3] action(::Symbol, ::Array{Union{Base.PkgId, Module},1}, ::Formatted; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:221
 [4] action at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:192 [inlined]
 [5] action(::Symbol, ::Array{Union{Base.PkgId, Module},1}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:181
 [6] action at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:181 [inlined]
 [7] loadstreaming(::String; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:113
 [8] loadstreaming(::String) at /home/karl/.julia/packages/FileIO/BUPwf/src/loadsave.jl:110
 [9] top-level scope at REPL[6]:1

Possible Solution / Implementation

I skipped over the source and it seems that FileIO is not among this packages dependencies and is also nowhere used? Are the implementations of Libz.load and Libz.loadstreaming actually missing?

Context

I am trying to load compressed FITS files using FITSIO.jl and this package. But the error happens with any Gzipped file.

Your Environment

This is thestatus of my test environment:

Status `/tmp/jl_uY0Ctd/Project.toml`
  [5789e2e9] FileIO v1.6.4
  [2ec943e9] Libz v1.0.1
kescobo commented 3 years ago

@karlwessel I can reproduce this, but the whole stacktrace is referring to FileIO, so I think that might be the best place to address this. Also, from the README:

NOTE: If you are starting a new project on Julia 0.6 or later, it is recommended to use the CodecZlib.jl package instead. CodecZlib.jl and other packages offer more unified interfaces for a wide range of file formats.

As an example, 

julia> using CodecZlib

julia> stream = GzipDecompressorStream(open("test.log.gz"))
TranscodingStreams.TranscodingStream{GzipDecompressor, IOStream}(<mode=idle>)

julia> for line in eachline(stream)
           println(line)
       end
┌ Warning: this is a test
│   x = 3
└ @ Main REPL[22]:2

Probably we should replace the entry for Libz.jl in FileIO. I've opened an issue to that effect: https://github.com/JuliaIO/FileIO.jl/issues/316

kescobo commented 3 years ago

@jakobnissen We should probably add this package to the list of packages for deprecation / archiving

karlwessel commented 3 years ago

Thanks for the quick verification.

I don't know how important the FileIO integration for gzipped files actually is. In my case, because of https://github.com/JuliaIO/FileIO.jl/issues/32, I thought FileIO could automatically handle compressed FITS files by first uncompressing them with Libz and then forwarding to the FITSIO loader. But I think I misread that issue and that behaviour isn't implemented yet anyway.

So not quite sure what the usecase for compression algorithms with FileIO really is, it is seldom that you want to "load" a compressed file.

Not to say that I don't think that the above behaviour should be fixed, either by implementing the loading correctly or by removing the registry entry, just that it is not such a pressing matter.

kescobo commented 3 years ago

Fair enough, worth correcting the reference, in any case. I don't use FileIO myself, so I'm not the best judge of usefulness.

xref #69.