search

pypa / bandersnatch

A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/
Academic Free License v3.0
455 stars 141 forks source link

bandersnatch script crashes with remaining 32448 items in TODO #958

Closed ksanwar closed 3 years ago

ksanwar commented 3 years ago

Hi, I am trying to create an air gapped repo. I have downloaded about 7TB of packages but the script keeps failing with memory error. Does bandersnatch maintain package dependencies? Do whatever packages I have downloaded so far have their dependencies downloaded as well? Trying to determine if my currently downloaded files are useable with all dependencies resolved.

thanks, Kazi

cooperlees commented 3 years ago

Hi,

What's the error you're getting, your config and your platform (operating system)? How much memory is available to bandersnatch?

Unfortunately, bandersnatch is not dependency aware. I was hoping to one day make a plugin using the new pip resolver but it seems to not have a friendly public API for this use.

If are you able to profile the memory to see what's being used could also possibly help us see if we can tune what's stored in memory too.

cooperlees commented 3 years ago

Please re-open if you ever want to give more info etc.