Caching mirrors

[brianjmurrell]

There are tools that allow one to mirror on demand as a caching proxy mirror. Such mirrors of course are virtually complete but in reality only as complete as the clients that use it have requested files/packages from it.

The problem with this is that report_mirror may report only a partial listing, but the reality is that the mirror is complete from the perspective of it's clients.

It would be useful if mirrormanager and/or report_mirror could be informed of this so that partial report_mirrors don't make invalidate the mirror. In the short-term report_mirror could be modified to fake the output of a full mirror, for example.

[adrianreber]

There is the flag Always up to date. We use it in our Fedora and CentOS Stream setups to mark primary mirrors and cloudfront based caching setups.

Does that help?

[brianjmurrell]

That might, if I could find where it's set. I've looked in both my site and host configuration and don't see that option.

[adrianreber]

That might, if I could find where it's set. I've looked in both my site and host configuration and don't see that option.

It is an admin only option.

[brianjmurrell]

Admin for the [private] mirror? I should be admin for my own private mirror. I can change many other aspects of my site and host.

[adrianreber]

Ah, private mirror. Right. No, it is a mirrormanager instance admin only option. For private mirrors you cannot set it yourself.

It doesn't sound problematic for private mirrors but currently it is not implemented.

[brianjmurrell]

So, back to original problem description. :-)

[adrianreber]

You could fake it by creating the directory tree with empty directories. report_mirror only scans directories (if I remember it correctly) to reduce I/O load. In theory a directory tree with no files should lead to an up to date mirror via report_mirror

[brianjmurrell]

Indeed. But what should all be in such an empty directory tree? And it does change from release to release (i.e. the introduction of Modular for example) so what would need to be created would need to be different on a per release basis.

This issue is all about resolving this problem given those constraints.

[brianjmurrell]

I suppose one could do something like:

# rsync -nai -f"+ */" -f"- *" rsync://<mirror>/fedora/releases/35 /path/to/local/mirror/fedora/releases/

To mimic the directory structure. Just sucks that this needs to be such a manual operation on every new release of a distro.