search

JuliaIO / JLD2.jl

HDF5-compatible file format in pure Julia
Other
553 stars 88 forks source link

Memory leak for large files #605

Open lassepe opened 1 week ago

lassepe commented 1 week ago

The following file created with JLD2 0.4.53 causes memory leaks on my system (Ubuntu 22.04, Julia 1.11.0-rc3):

https://drive.google.com/file/d/1_mjdRDD-DhrEsLoVy31sDis5sGpRo-mW/view?usp=sharing

Specifically, if I load the contained file as foo = load_object("training_data.jld2") and then do foo = nothing; GC.gc(true), the memory is never freed again. Hence, after a few consecutive loads, my julia sessions goes OOM.

JonasIsensee commented 1 week ago

What packages should I install to load this? The JLD2 fallback type reconstruction makes fewer performance optimizing assumptions making the loading very slow.

JonasIsensee commented 1 week ago

After some first tests, I see signifcant increase in (remaining) memory usage after the first load but it does not increase after that. The problem could be related to the fact that the current implementation for the MmapIO backend requires the full file size to be available as a contiguous chunk in memory. However, Julia will likely have a rather disorganized heap structure after the first loads. (lots of intermediate allocations and also new method compilation etc.)

lassepe commented 1 week ago

Sorry, yeah the package that this data originates from is not public. I can try to see if the same problem occurs with “random” data. I wonder whether this is a linux vs. windows issue due to differences in handling mmaping? The machine that this occurs on has 30GB RAM which is should be more than enough to find non-fragmented memory blocks for this. Another specialty of the machine this happened on is that it doesn’t have any swap.

JonasIsensee commented 5 days ago

I also tested on linux. Hm, lack of swap probably does change the flexibility of ram usage but I don't know.