change item in existing collection

[romainvieme]

Hi, I am using JLD2 to save the results of scientific computations and I wonder if there is a way to modify an array inside a JLD file. Here is an example of what I would like to do

using JLD2
myarray= [1 2 3]
@save "testfile.jld2" myarray
jldopen("testfile.jld2", "r+") do myfile
    myfile["myarray"][2]=4
end
jldopen("testfile.jld2", "r") do myfile
    myfile["myarray"]
end

and I would like to get [1 4 3] instead of the current [1 2 3]. I found that it is possible to make the change persist by opening the file with h5open but not with jldopen. Is there a way to make such persistent changes in JLD2 ?

[JonasIsensee]

Currently that is not possible but it would surely be nice to have.

[JonasIsensee]

Out of place modification is possible by deleting a dataset and writing to it again

A list of what is needed for this:

For modifying arrays:

• contiguous storage (the simplest case): allow mmap backed array
• compact storage: mmap backed (?), need to update checksum after write / at file close
• chunked storage: currently that means compression, so not possible to modify
• Allowed datatypes ? Anything should in principle be fine, turns out of place, if it contains references.

For scalars:

• Need notion of Dataset. JLD2 currently has no memory repr of a dataset. For manipulation, need Dataset struct that knows location in file, size and julia type. Need explicit, separate API for this kind of stuff. Likely try this is JLD2DebugTools. Constraint, need to keep julia type and everything the same
• Dictionaries: Cannot append to them, modifying values possible in principle but likely not worth the effort.

[koehlerson]

@JonasIsensee how would I delete the dataset? I'm trying to do the same right now as the author of the issue

edit: nvm its just the dictionary interface with delete!

[JonasIsensee]

As of version v0.4.53 Dataset objects support getindex and setindex! which should behave as the syntax suggests. This allows updating values of arrays.