wish: automatic cache pruning - Githubissues

kevincobain2000 / cache-http

action/cache temporary alternative to get dependency cache on GHES for self-hosted runners

https://medium.com/web-developer/github-actions-solving-actions-cache-v2-for-self-hosted-runners-on-github-enterprise-663f22caeee3

MIT License

17 stars 7 forks source link

wish: automatic cache pruning #9

Open markstos opened 2 years ago

markstos commented 2 years ago

To make this service more "just works", it could automatically prune the cache. Here's how that could work:

Right after a new upload completes and a success response has been returned to the client, A check could be made on the cache size or the number of items in the cache. For example, "If the number of the cached items exceeds 10, prune the cache". or "Prune items older than 10 days". This could possibly be it's own route that action-cache-http issues a "fire and forget" request to. (It issues the request but doesn't wait for the response).
Configuration could happen via the action-cache-http project. The pruning configuration would be passed to the /prune route if a custom pruning rule is needed.

kevincobain2000 commented 2 years ago

Normally, we prune it by cron tab. Diskspace is cheap, memory is expensive. Plus for around 20 projects, we stay in the range of 20GB ~ 50 GB. A crontab just takes care of those things.

Having such feature, would be a neat feature of course. Something like keep_days: 7.

markstos commented 2 years ago

Memory is expensive but if the cache service runs as a "service container", it's only running when it's needed.

kevincobain2000 commented 2 years ago

Or it can be done as

cache_http_api: "http://127.0.0.1:3000?keep_days=7"

This can be handled at the time of request

kevincobain2000 commented 2 years ago

https://github.com/actions/cache/issues/505#issuecomment-1123435993

markstos commented 2 years ago

@kevincobain2000 Do the recent changes to action/cache obsolete this project? I asked on the linked ticket, but I presume you linked that ticket here because those changes impact this project.

kevincobain2000 commented 2 years ago

Hi @markstos, yes if GHES can use action/cache upon 3.5 GHE release, then this project will become redundant. Although, of course the action/cache, will need to lift off the GHES check, which seems like they have already done it https://github.com/actions/cache/blob/136d96b4aee02b1f0de3ba493b1d47135042d9c0/src/utils/actionUtils.ts#L81

kevincobain2000 commented 2 years ago

Thanks for your https://github.com/actions/cache/issues/505#issuecomment-1126833743 @markstos I didn't know that the actions/cache is going to Github's cloud and not restrcited to GHES. That'd be a bummer since we have internal code in form of packages on internal pkg managers such as satis/artifactory that we wouldn't want to publish over the cloud.