Add function to emergency clear all old files from a repo

[puiterwijk]

Signed-off-by: Patrick Uiterwijk puiterwijk@redhat.com

[puiterwijk]

For reference: https://lists.fedoraproject.org/archives/list/infrastructure%40lists.fedoraproject.org/thread/SJVBJMVUKZJ6QQQWZ4UGFDY4DTDDQU2F/

[pypingou]

The idea being that the next pickle will be streamed off the old data then, which will make mirrorlist have only the 100% sure updated mirrors?

It is worth describing this in the doc of the script?

[puiterwijk]

I think the new docstring should be reasonable clear. Please let me know if it isn't.

[pypingou]

This looks all good to me, I think I was expecting some sort of flow:

• Mash the updates and ensure it's pushed to the master mirror
• Wait for a couple of hours
• Run this script
• Wait or run the pickle generation manually
• Propagate the new pickle to the MirrorList mirror
• Check it by doing...

What do you think?

[puiterwijk]

That's the flow that I wanted to enable indeed. I just didn't document the other steps, as I think that that's releng's responsibility and not part of what mirrormanager2 can do (and I'll add this when I write an SOP for them), or we have scripts for them already.

[adrianreber]

MirrorManager is confusing. At other places we are using category to differentiate between Fedora and EPEL. Here is it product. Not sure if this script should operate on products or categories.

And also not knowing which is the right approach, I would have expected to run the code at:

age_file_details

which is used to automatically remove older alternate entries. I would have expected just to remove all alternate entries for the repository/directory instead of deleting file objects from the database.

This does not mean I am against this PR, just stating how I expected this to be implemented.

[puiterwijk]

Note that I have done basically the same as age_file_details, except that I'm using SQLAlchemy ORM rather than raw SQL to do the quering. I'm still getting the files, and then removing all files with the same name that aren't the most recent (instead of what age_file_details does with propagation and stale delays).

Another option would probably be an age_file_details where you provide stale=1hour and propagation=1minute, but the advantage of this version is that it works on repositories, which means that we don't also clear files for other versions. (For example: if we trigger this for Fedora 23, we don't also want to clear EPEL7, that would be a big a waste of master mirror resources).

Also, the reason I'm using product to distinguish is that that's what Version is linked against, and has an easily enterable name: Fedora vs Fedora Linux. I also considered entering repo names, but those are even more cumbersome.

[adrianreber]

Thanks for the explanation. That sounds reasonable.

[adrianreber]

:+1: for merging

[puiterwijk]

Thanks!