Feature Request: Exclude Directories

[the-maldridge]

Over at @voidlinux we use mirrorbits to do mirror routing for our managed and unmanaged mirrors. Since using mirrorbits is becoming cost prohibitive for our package archives we'd like to exclude this directory, but still provide geolocated links for files that a human would manually find, such as ISOs and static toolsets.

This doesn't seem to be a feature currently, but I think it would be really useful and would allow using mirrorbits for subsets of mirrors. I envision this being a config token, likely in the form of a regex to match the paths to be excluded from the indexed set.

[etix]

I don't quite understand what would be the purpose of that feature. What would happen to excluded paths? Would they be served from the origin instead of a mirror? completely ignored by mirrorbits like if the file didn't exist?

[the-maldridge]

The purpose is to make certain directories be fully ignored by mirrorbits. Ideally mirrorbits would treat the file as though it doesn't exist and completely ignore it. The idea here is that it can help to reduce the memory usage of the associated redis instance when there are tens of thousands of files changing regularly which mirrorbits can't handle efficiently anyway.

In the specific case of Void Linux where I'd like to deploy this, we generally recommend that people set the package archive mirror by hand, but for files that we have links to such as ISOs, static tools, and documentation sets, it would be nice to still use mirrorbits to find a nearby copy of these assets. Changing the layout of the mirror to do this is possible, but the ability to just ignore files matching a particular regex would in my opinion be cleaner.

[etix]

Got it, thanks for the clarification. I understand that reshaping a repo is not always an option so I will think about the best way to achieve that.