search

JuliaIO / JLD2.jl

HDF5-compatible file format in pure Julia
Other
547 stars 85 forks source link

Internal limit of filename length? #424

Closed judober closed 1 year ago

judober commented 1 year ago

It seems that the filename length (at least on windows) is limited:

julia> jldsave("1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________.jld2"; a=1)
┌ Warning: Opening file with JLD2.MmapIO failed, falling back to IOStream
└ @ JLD2 C:\Users\judober\.julia\packages\JLD2\HnW0g\src\JLD2.jl:277
ERROR: SystemError: opening file "1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________.jld2": Invalid argument
Stacktrace:
  [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
    @ Base .\error.jl:176
  [2] #systemerror#80
    @ .\error.jl:175 [inlined]
  [3] systemerror
    @ .\error.jl:175 [inlined]
  [4] open(fname::String; lock::Bool, read::Bool, write::Bool, create::Bool, truncate::Bool, append::Bool)
    @ Base .\iostream.jl:293
  [5] openfile
    @ C:\Users\judober\.julia\packages\JLD2\HnW0g\src\JLD2.jl:267 [inlined]
  [6] openfile(T::Type, fname::String, wr::Bool, create::Bool, truncate::Bool, fallback::Type)
    @ JLD2 C:\Users\judober\.julia\packages\JLD2\HnW0g\src\JLD2.jl:278
  [7] jldopen(fname::String, wr::Bool, create::Bool, truncate::Bool, iotype::Type{JLD2.MmapIO}; fallback::Type{IOStream}, compress::Bool, mmaparrays::Bool, typemap::Dict{String, Any})
    @ JLD2 C:\Users\judober\.julia\packages\JLD2\HnW0g\src\JLD2.jl:334
  [8] jldopen(fname::String, mode::String; iotype::Type, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:compress,), Tuple{Bool}}})
    @ JLD2 C:\Users\judober\.julia\packages\JLD2\HnW0g\src\JLD2.jl:414
  [9] jldopen(::Function, ::String, ::Vararg{String}; kws::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:compress, :iotype), Tuple{Bool, DataType}}})
    @ JLD2 C:\Users\judober\.julia\packages\JLD2\HnW0g\src\loadsave.jl:2
 [10] #jldsave#79
    @ C:\Users\judober\.julia\packages\JLD2\HnW0g\src\loadsave.jl:243 [inlined]
 [11] top-level scope
    @ REPL[8]:1

caused by: SystemError: opening file "1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________1_________.jld2": Invalid argument
Stacktrace:
  [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
    @ Base .\error.jl:176
  [2] #systemerror#80
    @ .\error.jl:175 [inlined]
  [3] systemerror
    @ .\error.jl:175 [inlined]
  [4] open(fname::String; lock::Bool, read::Bool, write::Bool, create::Bool, truncate::Bool, append::Bool)
    @ Base .\iostream.jl:293
  [5] MmapIO
    @ C:\Users\judober\.julia\packages\JLD2\HnW0g\src\mmapio.jl:105 [inlined]
  [6] openfile
    @ C:\Users\judober\.julia\packages\JLD2\HnW0g\src\JLD2.jl:270 [inlined]
  [7] openfile(T::Type, fname::String, wr::Bool, create::Bool, truncate::Bool, fallback::Type)
    @ JLD2 C:\Users\judober\.julia\packages\JLD2\HnW0g\src\JLD2.jl:275
  [8] jldopen(fname::String, wr::Bool, create::Bool, truncate::Bool, iotype::Type{JLD2.MmapIO}; fallback::Type{IOStream}, compress::Bool, mmaparrays::Bool, typemap::Dict{String, Any})
    @ JLD2 C:\Users\judober\.julia\packages\JLD2\HnW0g\src\JLD2.jl:334
  [9] jldopen(fname::String, mode::String; iotype::Type, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:compress,), Tuple{Bool}}})
    @ JLD2 C:\Users\judober\.julia\packages\JLD2\HnW0g\src\JLD2.jl:414
 [10] jldopen(::Function, ::String, ::Vararg{String}; kws::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:compress, :iotype), Tuple{Bool, DataType}}})
    @ JLD2 C:\Users\judober\.julia\packages\JLD2\HnW0g\src\loadsave.jl:2
 [11] #jldsave#79
    @ C:\Users\judober\.julia\packages\JLD2\HnW0g\src\loadsave.jl:243 [inlined]
 [12] top-level scope
    @ REPL[8]:1

I increased the Path Length of Windows, however the error persists.

Is this an internal limitation of JLD2 (or FileIO) or am I doing something wrong? If this can't be fixed I would at least suggest to add a clearer error message.

JonasIsensee commented 1 year ago

You can see from the stacktrace that the error is triggered deep down. JLD2 asks julia to open a file and julia then asks the operating system to do this. The operating system returns an error.

I don't think julia / JLD2 can know in general why the operating system fails to open / create a file.

If you think this is a problem of julia, you can try to manually create a file on your computer with that filename. (to test if this is the problem).

[1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
    @ Base .\error.jl:176
  [2] #systemerror#80
    @ .\error.jl:175 [inlined]
  [3] systemerror
    @ .\error.jl:175 [inlined]
  [4] open(fname::String; lock::Bool, read::Bool, write::Bool, create::Bool, truncate::Bool, append::Bool)
    @ Base .\iostream.jl:293
judober commented 1 year ago

You are right. I tried it and it didn't work. Closing this as it seems there can be nothing done on the julia side.