lancachenet / monolithic

A monolithic lancache service capable of caching all CDNs in a single instance
https://hub.docker.com/r/lancachenet/monolithic
Other
725 stars 73 forks source link

Add support for easy cleaning #170

Closed VibroAxe closed 6 months ago

VibroAxe commented 1 year ago

A file can be "cleaned" from the cache using the below command from the host find /your/cache/folder -type f -exec awk 'FNR>2 {nextfile} /depot\/236391\/chunk\/cdd60ddacb14bd7d57e20c6972df2c1136d3e6f5/ { print FILENAME ; nextfile }' '{}' +

Add support to the docker container to allow easy cleaning using something similar to docker exec lancache_monolithic_1 clean <slug>

Da-Geek commented 11 months ago

Looks like you are already working on this. https://github.com/lancachenet/monolithic/pull/176

I have seen this awk example on https://lancache.net/docs/useful-commands/ and in a few comments around the lancache Discord #help-me channel. I was wondering why awk is being used in this case?

My awk is very very rusty, but I am trying to understand, why we even need the awk if we end up just deleting the file that match the query? If I am reading this correctly, the find is "finding/listing" every "file" within the cache directory structure and then running the awk against every file.

Why not just use grep and pipe the resulting list into rm? What am I missing that this process needs to use awk?

VibroAxe commented 11 months ago

The FNR>2 is important as it only searches the top two lines. You could head|grep&&rm but the awk is more efficient as you don't have the process overhead. Remember you have 65536 folders in a lancache environment, before you go into the numbers of files within a folder! Processing overhead on this command is important.

Da-Geek commented 11 months ago

The FNR>2 is important as it only searches the top two lines. You could head|grep&&rm but the awk is more efficient as you don't have the process overhead. Remember you have 65536 folders in a lancache environment, before you go into the numbers of files within a folder! Processing overhead on this command is important.

Thank You for explaining it, as I said "what am I missing?" So the awk is just looking at the "first 2 lines" of every file for a match to the thing we are seeing in the log...

so as not to have to escape so much within the awk I guess we just need to be a little simpler with what we pass it, to search for. rather than

find /your/cache/folder -type f -exec awk 'FNR>2 {nextfile} /depot\/236391\/chunk\/cdd60ddacb14bd7d57e2/ { rm FILENAME ; nextfile }' '{}' +

Could we just do:

find /your/cache/folder -type f -exec awk 'FNR>2 {nextfile} /236391/ { rm FILENAME ; nextfile }' '{}' +

as mentioned by @Lepidopterist

Just chucking some ideas around is all... Feel free to tell me to keep quiet :-D

stale[bot] commented 8 months ago

This issue has been automatically marked as inactive because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 6 months ago

This issue has been automatically closed after being inactive for 30 days. If you require further assistance please reopen the issue with more details or talk to us on discord