Out of memory with many files in batch mode

jonnor commented 4 years ago

Hi, thanks for making this implementation available.

Tested version: master as of April 15, 30abce1332b961b9b0234a22785e9de95fbcdb8e Machine platform. Arch Linux current, x86_64. Intel CPU, 16 GB RAM

Steps to reproduce:

Use visqol -batch_input_csv ... with a CSV with many pairs of audio. The first file I tested had 930 pair of samples a 5 seconds each, with samplerate 44.1kHz

Expected result

visqol will eventually complete all the files, write the CSV output and exit exit code 0

Actual result

visqol was killed Linux after 35 minutes, and processing around 750 files.

Retesting with a smaller subset of the data (45 files) the program completes successfully. But when looking at the memory usage of the process, it seems to grow linearly. Around 250 MB resident for 45 files.

Looking at the in VisqolManager::Run I see the AudioSamples being loaded with MiscAudio::LoadAsMono, containing an AMatrix with the audio data. But I do not see any destuctor in AudioSignal nor AMatrix, nor any manual cleanup of these after a sample pair has been processed.

Could it be that these are never freed?

mchinen commented 4 years ago

Thanks, I appreciate the details and investigation in your writeup. Indeed, this indicates some kind of memory leak, although at a glance it appears the default destructor should be applying to AudioSignal/AMatrix since they are not pointers. A while back we ran some runtime analysis on it to sort out memory leaks but I don't recall which conditions were tested. Let me know if you do look further into it.

mchinen commented 4 years ago

Just a note, when running (built with -c opt on 400 5s 16khz files the memory usage increased linearly, but 'only' got to 3.5% of my 64GB system at the end. Possibly this is related to the proto/debugging information being kept until it is output at the end to json/csv, which could be written to file in a streaming fashion instead. Not sure if this is the same issue though.

mchinen commented 4 years ago

Running on 1350 10s files brought me up to 12% of total memory on my system but did not crash. I think the streaming solution I mentioned would be better, but that seems like a separate issue. Please let me know if you see a crash again. Closing for now.

google / visqol

Out of memory with many files in batch mode #5