mkitti
commented
3 weeks ago Do you have a sample file available for testing?
Open hz-xiaxz opened 3 weeks ago
mkitti
commented
3 weeks ago Do you have a sample file available for testing?
JonasIsensee
commented
3 weeks ago EDIT: This is actually not really a problem with JLD2. The h5web viewer cannot cope with committed datatypes. (that are linked from groups) Demo:
I also opened an issue here: https://github.com/silx-kit/h5web/issues/1699
axelboc
commented
3 weeks ago We will definitely strive to support JLD2 files. Thanks for opening an issue in the H5Web repo. As explained, the problem comes from h5wasm not currently supporting committed datatypes. I've opened an issue on the h5wasm repo too: https://github.com/usnistgov/h5wasm/issues/80
loichuder
commented
2 days ago Thanks to @bmaranville and @axelboc work, we made good progress towards this. I could read this simple JLD2 file without issue using the main branch of H5Web: example.zip
# Inspired by https://github.com/JuliaIO/JLD2.jl?tab=readme-ov-file#jld2
using JLD2
jldsave("example.jld2"; x, y, z)
jldopen("example.jld2", "r+"; compress = true) do f
f["large_array"] = zeros(10000)
end
jldopen("example.jld2", "r+") do file
mygroup = JLD2.Group(file, "mygroup")
mygroup["mystuff"] = 42
end
JonasIsensee
commented
2 days ago I can confirm! Though, your example file here does not actually rely on the fix given that it does not use compound datatypes. The fact that the following file now works is quite impressive:
julia> using JLD2
julia> struct InnerStruct
x::String
y::Int
end
julia> struct OuterStruct
a::Int
b::InnerStruct
c::NTuple{3,Int}
end
julia> jldsave("nested_compound.jld2"; data=OuterStruct(1, InnerStruct("two",3),(4,5,6)))Thanks for double-checking :slightly_smiling_face:
I figured that the file was too simple so I hope to get a more complex example from people there: https://github.com/julia-vscode/julia-vscode/issues/2863#issuecomment-2340043098
JonasIsensee
commented
2 days ago Here are two more files which are conceptually interesting: JLD2 uses h5 references to refer to mutable fields inside structs and tracks object identities while saving and loading. This allows storing and loading recursive structures as well as obj identity preservation. In the second example, the field array is encoded and loaded only once. Both struct fields refer to the same memory after loading.
H5Web currently gives you no way to view / de-reference linked datasets in this way. That avoids all the potential pitfalls of circular references but also prevents viewing some of the data encoded in JLD2. Importantly though, it does not error even with files like this.
In my eyes, this is good enough for a release at this stage. Future feature ideas might be enhanced pretty printing for compound types and enabling the manual loading of referenced datasets.
julia> using JLD2
julia> mutable struct RecursiveStruct
x::Float64
y::RecursiveStruct
RecursiveStruct(x)=new(x)
RecursiveStruct(x,y)=new(x,y)
end
julia> jldsave("recursive.jld2"; r=^C
julia> r = RecursiveStruct(1)
RecursiveStruct(1.0, #undef)
julia> r2 = RecursiveStruct(2, r)
RecursiveStruct(2.0, RecursiveStruct(1.0, #undef))
julia> r.y = r2
RecursiveStruct(2.0, RecursiveStruct(1.0, RecursiveStruct(#= circular reference @-2 =#)))
julia> jldsave("recursive.jld2"; r)
julia> load("recursive.jld2")
Dict{String, Any} with 1 entry:
"r" => RecursiveStruct(1.0, RecursiveStruct(2.0, RecursiveStruct(#= circular … =#)))
julia> struct ObjIDPreservation
arr1::Vector{Int}
arr2::Vector{Int}
end
julia> arr = [1,2,3,4,5,6]
6-element Vector{Int64}:
1
2
3
4
5
6
julia> obj = ObjIDPreservation(arr, arr)
ObjIDPreservation([1, 2, 3, 4, 5, 6], [1, 2, 3, 4, 5, 6])
julia> obj.arr1 === obj.arr2 # references the same memory
true
julia> jldsave("objidpreservation.jld2"; obj)
julia> data = load("objidpreservation.jld2", "obj")
ObjIDPreservation([1, 2, 3, 4, 5, 6], [1, 2, 3, 4, 5, 6])
julia> data.arr1 === data.arr2
true
Is your feature request related to a problem?
JLD2is a file type natively created by Julia programming language, and it is heavily used in high performance scientific programs, JLD2JLD2is designed as comprising a subset of HDF5, though I'm non-expert in this repository, I think it might be reachable to supportJLD2with likewise interface.Alternatives you've considered
The authors of
JLD2.jlare eagering for a vscode-extension to visualizeJLD2files, see discussions below https://discourse.julialang.org/t/jld2-preview-in-vscode/80050/6 https://github.com/julia-vscode/julia-vscode/issues/2863Additional context
Currently opening a
JLD2using H5web gives error message like below, any hint to solve it?