search

pypa / bandersnatch

A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/
Academic Free License v3.0
455 stars 141 forks source link

Add/test `delete` + `verify` support for non 'filesystem' mirrors #554

Closed cooperlees closed 1 year ago

cooperlees commented 4 years ago

Delete and verify are not known to support anything other than filesystem well today. We should add swift and any other filesystem support and have a little more testing somehow.

LeoQuote commented 2 years ago

not sure if it's relevant, I'm using s3 storage to mirror packages, but the packages deleted in pypi.org was not deleted in my mirror sites. what can I do to troubleshoot this?

cooperlees commented 2 years ago

Sure is relevant. A mirror run has no idea what to delete. In order to work that out bandersnatch has to do a full sync and filesystem crawl to then compare what was synced and what was not to workout what to delete. Not an efficient or cheap process at all.

Without PyPI API enhancements, we don't have a good way here to workout what to delete periodically.

LeoQuote commented 2 years ago

Is there any existing command to achieve this? Or I have to implement it on my own?

cooperlees commented 2 years ago

This does not exist. Ideally to do this, we need to refactor verify (and delete) to use the storage abstractions to write to their respective back ends and mold the current state maintain by each sub command.

LeoQuote commented 1 year ago

Could this be considered done since https://github.com/pypa/bandersnatch/pull/1142 is merged?

cooperlees commented 1 year ago

Sure. I just haven't used it with non file system mirrors so didn't know if we'd covered it all. Happy to call it done due to your PR :)