On my system (Linux - Pop!_OS 20.04) running makePSID("user_input.json") works fine, but it uses a lot of memory (~10gb). However, for some reason the memory is never released after the function is done running. In fact at one point it actually crashed my computer
The memory usage increase is when the package reads in the data (unzip_data.jl)
datas = SortedDict(year => readPSID(filename) for (year, filename) in zip(years, filenames))
After running makePSID() I use 90% of my memory, when I had used ~30% before.
If I run the command a second time, sometimes it works, sometimes not. (i.e. sometimes I can finish building the datafile without Julia crashing and my laptop hanging).
I dont understand why the memory used in datas isnt released (or whatever the proper terminology is) after the function finishes.
What I have tried to do:
Added GC.gc() after some of the function calls in makePSID. Maybe this helped a little? It seemed to reduce my memeory use to ~80%. So clearly something else is still going on.
On my system (Linux - Pop!_OS 20.04) running
makePSID("user_input.json")works fine, but it uses a lot of memory (~10gb). However, for some reason the memory is never released after the function is done running. In fact at one point it actually crashed my computerThe memory usage increase is when the package reads in the data (
unzip_data.jl)makePSID()I use 90% of my memory, when I had used ~30% before.I dont understand why the memory used in
datasisnt released (or whatever the proper terminology is) after the function finishes.What I have tried to do:
GC.gc()after some of the function calls inmakePSID. Maybe this helped a little? It seemed to reduce my memeory use to ~80%. So clearly something else is still going on.