TreeIndex slow memory reclamation

[rsdy]

It seems like TreeIndex will leak significant amounts of memory with large number of insertions. I'm trying to insert a few billion records into a TreeIndex, with periodically cleaning it, like so:

    let index = scc::TreeIndex::new();

    for records in files
        .into_iter()
        .filter_map(|file| LogRecord::<MyStruct>::load_cbor(&file).ok())
        .flatten()
    {
        index
            .insert(ts.clone(), scc::ebr::Arc::new(record))
            .ok()
            .unwrap();

        records_written += 1;

        /// ... do something

        if records_written % 1_000_000 == 0 {
            index.clear();
            println!("records_written: {}", records_written);
        }
    }

I can observe the memory use keeps climbing even though i call clear() to try to control it.

I'm using 0.6 for these tests.

[wvwwvwwv]

Thanks for reporting this. I've briefly tested the scenario, and it indeed seems problematic. Actually, it is not a leak, instead, ebr reclaims garbage instances way too slowly. I'll try to make memory reclaim faster..

[wvwwvwwv]

It will take some time to fix the issue. Expediting memory reclamation is not as simple as I expected.

[wvwwvwwv]

The fix speeds memory reclamation up by a factor of 3; not quite satisfactory, that it is the best that I can get in short time. Hope that it will help. => Will be included in 0.6.1.

[wvwwvwwv]

more aggressive memory reclamation applied. => Will be included in 0.6.2.

[wvwwvwwv]

Yet another node reclamation method will be implemented in 0.7.0.