mkitti
commented
3 months ago Do you have a sample file available for testing?
Closed hz-xiaxz closed 2 months ago
mkitti
commented
3 months ago Do you have a sample file available for testing?
JonasIsensee
commented
3 months ago EDIT: This is actually not really a problem with JLD2. The h5web viewer cannot cope with committed datatypes. (that are linked from groups) Demo:
I also opened an issue here: https://github.com/silx-kit/h5web/issues/1699
axelboc
commented
3 months ago We will definitely strive to support JLD2 files. Thanks for opening an issue in the H5Web repo. As explained, the problem comes from h5wasm not currently supporting committed datatypes. I've opened an issue on the h5wasm repo too: https://github.com/usnistgov/h5wasm/issues/80
loichuder
commented
2 months ago Thanks to @bmaranville and @axelboc work, we made good progress towards this. I could read this simple JLD2 file without issue using the main branch of H5Web: example.zip
# Inspired by https://github.com/JuliaIO/JLD2.jl?tab=readme-ov-file#jld2
using JLD2
jldsave("example.jld2"; x, y, z)
jldopen("example.jld2", "r+"; compress = true) do f
f["large_array"] = zeros(10000)
end
jldopen("example.jld2", "r+") do file
mygroup = JLD2.Group(file, "mygroup")
mygroup["mystuff"] = 42
end
JonasIsensee
commented
2 months ago I can confirm! Though, your example file here does not actually rely on the fix given that it does not use compound datatypes. The fact that the following file now works is quite impressive:
julia> using JLD2
julia> struct InnerStruct
x::String
y::Int
end
julia> struct OuterStruct
a::Int
b::InnerStruct
c::NTuple{3,Int}
end
julia> jldsave("nested_compound.jld2"; data=OuterStruct(1, InnerStruct("two",3),(4,5,6)))Thanks for double-checking :slightly_smiling_face:
I figured that the file was too simple so I hope to get a more complex example from people there: https://github.com/julia-vscode/julia-vscode/issues/2863#issuecomment-2340043098
JonasIsensee
commented
2 months ago Here are two more files which are conceptually interesting: JLD2 uses h5 references to refer to mutable fields inside structs and tracks object identities while saving and loading. This allows storing and loading recursive structures as well as obj identity preservation. In the second example, the field array is encoded and loaded only once. Both struct fields refer to the same memory after loading.
H5Web currently gives you no way to view / de-reference linked datasets in this way. That avoids all the potential pitfalls of circular references but also prevents viewing some of the data encoded in JLD2. Importantly though, it does not error even with files like this.
In my eyes, this is good enough for a release at this stage. Future feature ideas might be enhanced pretty printing for compound types and enabling the manual loading of referenced datasets.
julia> using JLD2
julia> mutable struct RecursiveStruct
x::Float64
y::RecursiveStruct
RecursiveStruct(x)=new(x)
RecursiveStruct(x,y)=new(x,y)
end
julia> jldsave("recursive.jld2"; r=^C
julia> r = RecursiveStruct(1)
RecursiveStruct(1.0, #undef)
julia> r2 = RecursiveStruct(2, r)
RecursiveStruct(2.0, RecursiveStruct(1.0, #undef))
julia> r.y = r2
RecursiveStruct(2.0, RecursiveStruct(1.0, RecursiveStruct(#= circular reference @-2 =#)))
julia> jldsave("recursive.jld2"; r)
julia> load("recursive.jld2")
Dict{String, Any} with 1 entry:
"r" => RecursiveStruct(1.0, RecursiveStruct(2.0, RecursiveStruct(#= circular … =#)))
julia> struct ObjIDPreservation
arr1::Vector{Int}
arr2::Vector{Int}
end
julia> arr = [1,2,3,4,5,6]
6-element Vector{Int64}:
1
2
3
4
5
6
julia> obj = ObjIDPreservation(arr, arr)
ObjIDPreservation([1, 2, 3, 4, 5, 6], [1, 2, 3, 4, 5, 6])
julia> obj.arr1 === obj.arr2 # references the same memory
true
julia> jldsave("objidpreservation.jld2"; obj)
julia> data = load("objidpreservation.jld2", "obj")
ObjIDPreservation([1, 2, 3, 4, 5, 6], [1, 2, 3, 4, 5, 6])
julia> data.arr1 === data.arr2
true
hz-xiaxz
commented
2 months ago Sorry I don't really know how to use the main branch version of h5web, but I think tests above is good for most jld2 file case I will use
axelboc
commented
2 months ago I've started a discussion thread in the H5Web repo with improvement ideas mentioned in this issue so as to not lose track of them. Feel free to continue the discussion and/or create proper feature requests for these ideas over there.
Is your feature request related to a problem?
JLD2is a file type natively created by Julia programming language, and it is heavily used in high performance scientific programs, JLD2JLD2is designed as comprising a subset of HDF5, though I'm non-expert in this repository, I think it might be reachable to supportJLD2with likewise interface.Alternatives you've considered
The authors of
JLD2.jlare eagering for a vscode-extension to visualizeJLD2files, see discussions below https://discourse.julialang.org/t/jld2-preview-in-vscode/80050/6 https://github.com/julia-vscode/julia-vscode/issues/2863Additional context
Currently opening a
JLD2using H5web gives error message like below, any hint to solve it?