microsoft / FASTER

Fast persistent recoverable log and key-value store + cache, in C# and C++.
https://aka.ms/FASTER
MIT License
6.29k stars 563 forks source link

Full checkpoint deletes only one log file #881

Closed AqlaSolutions closed 9 months ago

AqlaSolutions commented 11 months ago

I have 74 hlog.log.* files 1GB each. When I do this:

        _fasterKv.Log.ShiftBeginAddress(_fasterKv.Log.HeadAddress);
        _fasterKv.TryInitiateFullCheckpoint(out _, CheckpointType.FoldOver);

it deletes only one log file per checkpoint but others stay.

How can I force it to delete everything else?

AqlaSolutions commented 11 months ago

It looks like checkpoint state machine hangs and never changes from 18,38 to 21,37 state

AqlaSolutions commented 11 months ago

I tried calling TakeFullCheckpointAsync instead and it completed in seconds but no log files were removed at all this time. image

image

TruncateLog does nothing: image

AqlaSolutions commented 11 months ago

I ended up writing a code to automatically remove files with segment number < BeginAddress or > Head or Tail Address segments (after performing checkpoints).

badrishc commented 9 months ago

I'm not sure why it isn't deleting for you, works fine in our deployments. Feel free to reopen if you have a stand alone repro.