Open NightFury13 opened 7 years ago
the memory should be freed if you set your variable to nil and collectgarbage() for example. Is there a small test case that reproduces this that I can investigate?
This should act as a dummy test case for what I'm doing...
[READ AS : th> torch-code || mem> CPU-RAM being used (output from htop
)]
th>
mem> 8900
th> tds = require 'tds'
mem> 9312
th> a = tds.Hash()
mem> 9312
th> for i=1,800 do
..> b = tds.Hash()
..> for j=1,32 do
..> b[j] = 'something'
..> end
..> a[i] = b
..> end
mem> 12672
th> a = nil
mem> 12672
th> b = nil
mem> 12672
th> collectgarbage()
mem> 11224
I see that not all the memory is being given back to me (11224---8900) once I re-allocate these variables. So if you consider the above code-snippet as a 800x32 dimensional feature, I have a LOT of such features and this wastage of space for each feature ends up using up my complete RAM even if I use the same variables and garbage-collection to load the data in chunks.
@soumith were you able to reproduce this problem?
@NightFury13 first of all, modern allocators dont always release all the memory back to the system unless needed to do so. They might keep the memory for a while. Second, I reproduced your problem at a small scale (like in your example), but not at a large scale. After a certain amount of memory limit, the process just gives back the memory to the OS.
Can you create a dummy snippet of code that generates 800x32 dimensional features. I want a snippet that will really get me to using a LOT of ram, ideally gigabytes of ram.
How does one explicitly free the memory assigned to any variable which is an instance of tds.Hash()?
I have quite some data that I want to run my torch codes on.However, I face CPU-RAM limitations even if I load my data in chunks as the memory occupied by previously allocated tds is not being freed up even after setting the
variable to nil
and doing acollectgarbage()
in the code.