Memory Usage for Large File Operations

[XlousZ]

{
    flare::NoncontiguousBuffer standard_output;
    ......
    auto result = flare::FlattenSlow(output->standard_output);
    auto deps_list = flare::Split(result, '\n');
}

a 400 MB file is cached in standard_output. After I use it, I find that the memory is still occupied and cannot be released.

[0x804d8000]

NoncontiguousBuffer is implemented using object pool, memory reclamation may be deferred in certain cases.

By default, several hundreds megabytes memory will be reserved by the object pool. See if memory usage keeps increasing when you cache and free more files.

[XlousZ]

Reaching %5 and no more increasing. Fortunately, my data can be processed by line. Reading data by row to avoid memory usage of large file data.