JuliaIO / JLD2.jl

HDF5-compatible file format in pure Julia
Other
547 stars 85 forks source link

@load causes an error when file is missing *even when the line is not executed* #430

Closed glwhart closed 1 year ago

glwhart commented 1 year ago

This is so strange, but I'm getting an error when @load refers to a missing file...but this happens even when the line with @load is never executed. Here is a minimal working example:

using JLD2
i = 1
if true
    @save "myvars"
else
    @load "myvars"
end

When I save this as a script and call it with julia script.jl I get an error:

(base) ~/doubleDescent> julia script.jl 
┌ Warning: Opening file with JLD2.MmapIO failed, falling back to IOStream
└ @ JLD2 ~/.julia/packages/JLD2/Pi1Zq/src/JLD2.jl:240
ERROR: LoadError: SystemError: opening file "myvars": No such file or directory
Stacktrace:
  [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
    @ Base ./error.jl:174
  [2] #systemerror#68
    @ ./error.jl:173 [inlined]
  [3] systemerror
    @ ./error.jl:173 [inlined]
  [4] open(fname::String; lock::Bool, read::Bool, write::Bool, create::Bool, truncate::Bool, append::Bool)
    @ Base ./iostream.jl:293
  [5] openfile
    @ ~/.julia/packages/JLD2/Pi1Zq/src/JLD2.jl:230 [inlined]
  [6] openfile(T::Type, fname::String, wr::Bool, create::Bool, truncate::Bool, fallback::Type)
    @ JLD2 ~/.julia/packages/JLD2/Pi1Zq/src/JLD2.jl:241
  [7] jldopen(fname::String, wr::Bool, create::Bool, truncate::Bool, iotype::Type{JLD2.MmapIO}; fallback::Type{IOStream}, compress::Bool, mmaparrays::Bool, typemap::Dict{String, Any})
    @ JLD2 ~/.julia/packages/JLD2/Pi1Zq/src/JLD2.jl:297
  [8] jldopen(fname::String, wr::Bool, create::Bool, truncate::Bool, iotype::Type{JLD2.MmapIO})
    @ JLD2 ~/.julia/packages/JLD2/Pi1Zq/src/JLD2.jl:260
  [9] jldopen(fname::String, mode::String; iotype::Type, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ JLD2 ~/.julia/packages/JLD2/Pi1Zq/src/JLD2.jl:363
 [10] jldopen (repeats 2 times)
    @ ~/.julia/packages/JLD2/Pi1Zq/src/JLD2.jl:358 [inlined]
 [11] var"@load"(__source__::LineNumberNode, __module__::Module, filename::Any, vars::Vararg{Any})
    @ JLD2 ~/.julia/packages/JLD2/Pi1Zq/src/loadsave.jl:133
in expression starting at /home/glh43/doubleDescent/script.jl:6
in expression starting at /home/glh43/doubleDescent/script.jl:3

caused by: SystemError: opening file "myvars": No such file or directory
Stacktrace:
  [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
    @ Base ./error.jl:174
  [2] #systemerror#68
    @ ./error.jl:173 [inlined]
  [3] systemerror
    @ ./error.jl:173 [inlined]
  [4] open(fname::String; lock::Bool, read::Bool, write::Bool, create::Bool, truncate::Bool, append::Bool)
    @ Base ./iostream.jl:293
  [5] JLD2.MmapIO(fname::String, write::Bool, create::Bool, truncate::Bool)
    @ JLD2 ~/.julia/packages/JLD2/Pi1Zq/src/mmapio.jl:105
  [6] openfile
    @ ~/.julia/packages/JLD2/Pi1Zq/src/JLD2.jl:233 [inlined]
  [7] openfile(T::Type, fname::String, wr::Bool, create::Bool, truncate::Bool, fallback::Type)
    @ JLD2 ~/.julia/packages/JLD2/Pi1Zq/src/JLD2.jl:238
  [8] jldopen(fname::String, wr::Bool, create::Bool, truncate::Bool, iotype::Type{JLD2.MmapIO}; fallback::Type{IOStream}, compress::Bool, mmaparrays::Bool, typemap::Dict{String, Any})
    @ JLD2 ~/.julia/packages/JLD2/Pi1Zq/src/JLD2.jl:297
  [9] jldopen(fname::String, wr::Bool, create::Bool, truncate::Bool, iotype::Type{JLD2.MmapIO})
    @ JLD2 ~/.julia/packages/JLD2/Pi1Zq/src/JLD2.jl:260
 [10] jldopen(fname::String, mode::String; iotype::Type, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ JLD2 ~/.julia/packages/JLD2/Pi1Zq/src/JLD2.jl:363
 [11] jldopen (repeats 2 times)
    @ ~/.julia/packages/JLD2/Pi1Zq/src/JLD2.jl:358 [inlined]
 [12] var"@load"(__source__::LineNumberNode, __module__::Module, filename::Any, vars::Vararg{Any})
    @ JLD2 ~/.julia/packages/JLD2/Pi1Zq/src/loadsave.jl:133

If I comment out line 6:

using JLD2
i = 1
if true
    @save "myvars"
else
    #@load "myvars"
end

the script runs without error, creates the file myvars. Then if I uncomment line 6 and run again, the script runs without error, and (re)writes "myvars" as expected.

JonasIsensee commented 1 year ago

Hi @glwhart,

I'm not a big fan of @load for this reason. (I removed all mentions to it in the readme about 2 years ago)

@load is a macro and therefore runs at parse-time. It will attempt to open the file and read the variable names to be loaded at parse time. It should work, if you just write @load "myvars" i or, well i = load("myvars", "i")

glwhart commented 1 year ago

Is there a better way to read in all the saved variables? Or is that just a bad thing to do?

I'm running an analysis script on remote cluster where all the data is. It's convenient sometimes to save the state of the script on the remote machine and then copy the state locally to debug the script or do further analysis.

JonasIsensee commented 1 year ago

Hm, that's an interesting question.

I honestly don't know a good solution to do this without specifying the variables that you want to load into the workspace.

glwhart commented 1 year ago

I think the best thing to do is to decide which variables to save and then just do a jldsave.