search

JuliaIO / JLD2.jl

HDF5-compatible file format in pure Julia
Other
547 stars 85 forks source link

compression of large matrix #425

Closed aurelio-amerio closed 1 year ago

aurelio-amerio commented 1 year ago

Hello, I have found a potential bug while saving a large matrix on disc with compression. I am using Julia 1.8.2. The code to reproduce the bug is the following:

using JLD2
using CodecZlib

data = Dict()
emptym = zeros(Float32, 87,12*1024^2)
data2["matrix"] = emptym

save("test.jld2", data, compress=true)

You will get a UInt32 truncation error. If you don't use compression, the data will be written correctly on disc. Could you please investigate this error? Thank you very much

JonasIsensee commented 1 year ago

Hi @aurelio-amerio ,

this is not really an issue in JLD2 and has been previously reported in #399. CodecZlib cannot currently compress arrays larger than 4GB. There is a fix in https://github.com/JuliaIO/CodecZlib.jl/pull/62 but it has not yet been merged.