search

JuliaIO / JLD2.jl

HDF5-compatible file format in pure Julia
Other
549 stars 85 forks source link

error when saving and loading SharedArray #392

Closed thudjx closed 11 months ago

thudjx commented 2 years ago

Hello, I am tryging to save and load a SharedArrays.SharedArray object into .jld2 file. The data is successfully saved, without any waring or error. But when I load it, errors ocurr:

julia> load("reEgens.jld2")
Error encountered while load FileIO.File{FileIO.DataFormat{:JLD2}, String}("reEgens.jld2").

Fatal error:
ERROR: ConcurrencyViolationError("setfield!: atomic field cannot be written non-atomically")

I am wondering if I am doing everything the right way. I have tried to first convert the SharedArray into an Array object and get the expected results, but I think this is the last thing I would do.

JonasIsensee commented 2 years ago

Hi @thudjx, I suppose, the disappointing answer is that shared arrays are useful for multiprocessing but not for storage. There is currently no support for them in JLD2. If you do not want to convert the whole thing in to an Array at once, you could consider writing chunks of the array separately.