search

tuwien-musicir / rp_extract

Rhythm Pattern music feature extractor by IFS @ TU-Vienna
GNU General Public License v3.0
110 stars 27 forks source link

massive memory consumption in rp_extract_batch #15

Closed audiofeature closed 8 years ago

audiofeature commented 8 years ago

when batch extracting features with rp_extract_batch the consumed main memory goes quickly up to several GB, including the system starting to swap.

find a way to free up memory in each for loop when the features are written and the file is processed

slychief commented 8 years ago

I have similar problems with long audio files (>60min). How long are the tracks in your collections?

audiofeature commented 8 years ago

some are long like this, but many are not, the memory usage still seems to increase with each file

do you know how to free memory? or when i assign a variable like this, when / how is the "lost" memory released? (just tiny random example)

for i ... RP = rp_extract(...) RP_SUM = some_aggregation(...) ALL_RPs.extend (RP_SUM)

Will all RP variables stay in memory?

Am 04.01.2016 um 11:38 schrieb Alexander Schindler:

I have similar problems with long audio files (>60min). How long are the tracks in your collections?

— Reply to this email directly or view it on GitHub https://github.com/tuwien-musicir/rp_extract/issues/15#issuecomment-168641394.

Thomas Lidy Vienna University of Technology Institute of Software Technology and Interactive Systems Favoritenstraße 9-11/188 A-1040 Vienna

http://www.ifs.tuwien.ac.at/~lidy http://musicbricks.net/

audiofeature commented 8 years ago

Added a gc.collect() call in ed5e51a which calls the garbage collector after extraction of each file and this seems to have solved the problem and keeps the memory consumption at a stable level.