monstermuffin / mergerfs-cache-mover

Python script for moving files on a cached disk to a backing pool with mergerFS
GNU General Public License v3.0
15 stars 5 forks source link

[Feature] Move from cache based on age #4

Open tigattack opened 2 months ago

tigattack commented 2 months ago

Currently, cache_mover will move files from cache to the disk pool when cache usage is over the configured threshold.

This is fine, but some use cases may result in not hitting the threshold for a long time, in turn resulting in data sitting in cache for a long time.

For example, I've not written much to my storage for a few weeks, but Frigate is constantly writing a rolling ~180GB of data. This results in ~20% usage on my cache disk, so none of this is never moved to the pool.

An age threshold that acts secondarily to the utilisation threshold would be useful for such cases. This logic could be disabled in config with e.g. AGE_THRESHOLD: 0.

Not a huge priority for me so no rush, and I'll likely get round to implementing this myself if you don't have time as IIRC I had some other tweaks I was planning to PR as well.

monstermuffin commented 2 months ago

I see.

Would this be an option to use instead of space used or were you imagining both would be used in tandem?

Also, if you are using 180GB of your cache which is 20%, is there any need for this to be moved to backing storage?

Frigate in your use case is deleting the old files anyway (assuming by rolling you mean it can only use 180GB as a limit?) so moving stuff to backing doesn't sound like it would do much but be deleted when Frigate writes a new files to cache and deletes an old one to adhere to the storage limit.

tigattack commented 2 months ago

Yeah, the example I gave isn't a great example tbh as it doesn't really prove the need for this for the reasons you've stated.

I still think it would be a useful option to have, and my idea was it would work in tandem with the capacity limit, i.e. if not at capacity limit, move on and check age limit if set.