search

ngryman / stores

Stores things efficiently. Don't worry about cache stampede anymore.
1 stars 0 forks source link

Cache expiration #4

Closed vorg closed 7 years ago

vorg commented 7 years ago

Currently the cache keeps growing indefinitely.

ngryman commented 7 years ago

Hey, it's been a long time I haven't touched this project so I would need a basic use case illustrating this and more precisions. Thanks!

vorg commented 7 years ago

I have a API that serves live data. The client app requests week of data in 24h chunks. As it takes 10s to prepare one chunk of 24h data we are catching it. After 6 months of operation we run out of HDD space due to amount of data we have cached. Due to balancing of the cache folder tree structure It's not easy to just delete "first (oldest) 10 files" every now and then to keep the space. Do you have any other suggestion how to have expiration data or a rolling cache that doesn't grow indefinitely?

ngryman commented 7 years ago

Ok I understand your issue now.

Well, the provided FileStore does not include any cache replacement facility (LRU, LFU, ..). stores only focuses on caching things efficiently, the way you manage your cache replacement logic is left to you.

Several solution exist to your problem:

  1. You can extend FileStore and add your own cache replacement logic here.
  2. You can create your own store via custom stores.
  3. You can build cron script that will collect all files, order them by creation date, keep the first nth and flush the rest.

Tell me the solution you prefer and I'll "try" to help you implement this (given the limited time I have lately 😂).

vorg commented 7 years ago

I think I'll go with option number 3 and seems like the simplest one

ngryman commented 7 years ago

Alright, let me know if you have trouble with this and if I can help. I'm closing the issue.

vorg commented 7 years ago

Just for the future reference here is the command i ended up with:

#remove files older than 30 days
find cache/ -type f -mtime +30 -exec rm {} \;