Initial notes/dump of ideas

[yarikoptic]

Motivation - complement https://snapshots.debian.org and https://snapshot-neuro.debian.net so we could

• say "pip, please use PyPI as of this specific DATE (e.g. as month ago)"
• make containers even more reproducible (reuse the date we are giving to nd_freeze)
• be able to reconstruct past environments (gimme env like I did run tests 1 month ago)

While chatting with @asmacdo , he mentioned various solutions which could let us accomplish the drill and I want to make note of them

• feasibility: according to https://pypi.org/stats/ there is "only" 13TB of storage needed so far
• https://pulpproject.org/pulp-workflow-overview/ is the most flexible platform to establish smth like "daily releases" of PyPI. Sounds similar to snapshot.debian.org in principle of function (farm of blobs etc)
  • https://www.redhat.com/en/technologies/management/satellite#? from RedHat might be based on it
• https://pypi.org/project/bandersnatch/ is for a complete "backup" of PyPI.
  • might want to configure to not remove anything
  • would still need to have some way to generate "per day" state of the PyPI from current point back (based on releases metadata) and then going forward
• first step might need downgrade (or upgrade) of pip itself to the version which was available on that past date so to match tooling and archive exactly

attn @jwodder if he knows more of other relevant solutions

[jwodder]

Pre-existing solution: https://github.com/astrofrog/pypi-timemachine

[yarikoptic]

Great find @jwodder, I have posted question there to see if it could indeed be the complete solution for my greedy evil plans ;-)

[asmacdo]

This looks promising. We would still need to host a pypi mirror to be fully complete.

pypi-timemachine would also need a feature to allow customizing the API endpoint (hardcoded to pypi.org) https://github.com/astrofrog/pypi-timemachine/issues/8

the request would look like:

pip -> filtering proxy (pypi-timemachine) -> mirror (bandersnatch)

IMO, allowing the cutoff date and python repo to be customized looks pretty minimal.

One thing to keep an eye on is that pypi-timemachine relies on part of the API from PEP 503, which is excluded from PEP 691. AIUI, warehouse has no plans to remove this from the pypi API though.