[Feature Request] Need an background GC worker to faster space reclaim

[royguo]

Currently, ZenFS doesn't re-use the existing zone unless all of its data is deleted by RocksDB.

Sometimes we may run into such case:

• Lots of zones have 95% deleted data but still have 5% valid data
• ZenFS is not going to re-claim those zones, thus space is wasted, and serving ability is restricted
• DBAs cannot monitor disk free space without change the monitor tool

We propose to add a new GC worker thread that does this:

• Periodically run in the background to check the zones with the same lifetime_hit
• Merge zones that only have a very small portion of valid data (e.g. 5% valid data, 90% deleted data)

Since this GC worker only used as a backup plan to avoid the worst cases, so there should be no harm to the ZenFS's performance first principle.

[yhr]

Garbage collection adds a lot of code complexity, so I've actively avoided implementing it. I'm trying to keep zenfs super-simple. Do you have any (estimated?) numbers on decreased space amplification in a real-world use case?

[royguo]

@yhr Now since you are going to implement a new background reset/finish worker, I believe it will be much easier for you to also add a GC ratio in that background worker, :P.

We can simply add a garbage ratio in the background reset/finish worker, then if we find out a zone has only x% valid data, we move it into another zone & reset the previous one). This process could be co-designed with the new background allocator.

[yhr]

@royguo : Yes, exactly! once #36 is done, lets revisit this.