microsoft / FASTER

Fast persistent recoverable log and key-value store + cache, in C# and C++.
https://aka.ms/FASTER
MIT License
6.29k stars 563 forks source link

After dispose and restore obj.log is growing always #882

Open Kosmikos opened 11 months ago

Kosmikos commented 11 months ago

Hi. I use FASTER as key-value cache with restore it after re-run application. And I have a problem with disk space. Steps to reproduce

  1. Create store
    var config = new FasterKVSettings<Key, Value>(baseDir)
    {
        TryRecoverLatest = true,
        RemoveOutdatedCheckpoints = true,
        MemorySize = 1L << 10,
        PageSize = 1L << 9,
        EqualityComparer = new Key.Comparer()
    };
    var store = new FasterKV<Key, Value>(config);
  2. Upsert values to store
  using var session = store.NewSession(new SimpleFunctions<Key, Value>());
    var random = new Random();
    for (var ii = 0; ii < 100; ii++)
    {
        var key = new Key(ii % keyMod);
        var value = new Value(key.key + random.Next(keyMod));
        session.Upsert(key, value);
    }
  1. Take checkpoin and dispose all
    store.TakeHybridLogCheckpointAsync(CheckpointType.Snapshot).GetAwaiter().GetResult();
    store.Dispose();
    config.Dispose();
  2. Repeat step 1->2->3.

Result File hlog.obj.log.0 grow after each iteration. And we confised because in each iteration we upsert same keys.

I read documentation about checkpoint and do not found how fix it.

This is .gif about problem SimpleChecks_j9jEP3DJRR

This is repo with code with promlem https://github.com/Kosmikos/FASTERLogGrowth

Please explain what I'm doing wrong

maoxianfly commented 2 months ago

You can try to use store.CheckpointManager.PurgeAll();

Kosmikos commented 2 months ago

Hello. Where should I add this line? I added here

    store.CheckpointManager.PurgeAll();
    store.TakeHybridLogCheckpointAsync(CheckpointType.Snapshot).GetAwaiter().GetResult();    
    store.Dispose();

Behavior has not changed