Wrong saving/reloading of Array in Julia 1.7.2

[bgctw]

I cannot store and re-read a simple array:

julia> import JLD2

julia> a = rand(Float64, (750, 39, 8)); a[1:10,1,1]
10-element Vector{Float64}:
 0.24759061931436066
 0.2727125191657097
 0.6034494487576191
 0.7876102956100266
 0.06975699358854048
 0.6821794798424281
 0.7781693345477846
 0.5100902753151138
 0.41144668498774284
 0.22806825786967144

julia> JLD2.save_object("tmp/tmp2.jld2",  a)

julia> a2 = JLD2.load_object("tmp/tmp2.jld2");

julia> a2[1:10,1,1]
10-element Vector{Float64}:
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0

This happens in a temporary environment where only added JLD2:

julia> Pkg.status()
      Status `/tmp/jl_DZYH2o/Project.toml`
  [033835bb] JLD2 v0.4.22

julia> VERSION
v"1.7.2"

I cannot reproduce this error on my local machine. It happens on a remote linux server at work with more memory, but where I have only limited rights. Are there requirements of system libraries, that I can request the admins to check?

[JonasIsensee]

Hi @bgctw,

This is very odd. I've never seen this before and not sure how this could happen. (Can't reproduce it either)

You could try a few things:

• Write the file on the remote machine and copy it to your local computer to see if the file is broken (or the loading).
• Write/Read the file using the IOStream backend to test if memory mapping might be acting up
• Write to a local folder that is not /tmp

[bgctw]

Thanks, indeed the file location makes a difference. In my code above I did not write to /tmp but to a local tmp dir (no slash). However, I am starting julia from a directory in a shared file system. If I replace the filename with a location that points to the local harddrive attached to the server, JLD2 works as expected.

When I write the file on the server (shared file system) and copy and open it on my local machine, it reads the problemenatic zeros. Hence, the problem seems to be with writing the file.

How do I write using the IOStream backend? does not work: JLD2.jldsave{IOStream}(fname1; a)

A colleague told me, that he encountered problems using mmap on the server.

You may ignore my frustration: The issue is, that such wrong behavior of JLD2 (or mmap) can go unnoticed without a warning. I could solve my original problem by replacing JLD2 with julia serialization. Using the local scratch is not an option, I need the shared file system. This behavior caused me so much time of debugging at the wrong place so that I cannot trust it any more. It lends evidence for the recent critics of Julia being not reliable.

[JonasIsensee]

does not work: JLD2.jldsave{IOStream}(fname1; a) Thanks for pointing this out. This is an error in the documentation. I will fix it right away.

jldsave(fn, IOStream; a)

A colleague told me, that he encountered problems using mmap on the server.

I'm sorry you are experiencing these issues. mmap is known to be difficult on shared file systems. However, there is nothing that JLD2 can do about this, as mmap is essentially provided by the operating system. (If this is reproducible, you could consider contacting your system administrator)