Sync only python 2 packages

[joeboughazali]

I'm trying to sync only packages compatible with Python 2 but it's not working

[mirror]
directory = /data/MINI_PYPI/bandersnatch/packages/
master = https://pypi.org
timeout = 20
workers = 3
hash-index = false
stop-on-error = false
json = true

[plugins]
enabled =
        allowlist_project
        regex_project_metadata
        blocklist_project
        exclude_platform
        latest_release

[allowlist]
packages =
        setuptools

[blocklist]
platforms =
        windows
        macos
        freebsd
        py3
        py3.0
        py3.1
        py3.2
        py3.3
        py3.4
        py3.5
        py3.6
        py3.7
        py3.8
        py3.9
        py3.10
        py3.11

[latest_release]
keep = 3

[regex_project_metadata]
any:info.classifiers =
        .*Development Status :: *
        .*Intended Audience :: *
        .*License :: *
        .*Operating System :: *
        .*Topic :: *

not-null:info.classifiers =
        .*Programming Language :: Python :: 2.*

[cooperlees]

What is happening? I'm guessing it's downloading lots of python3 only packages you don't want? Describing that is always an advantage when you open an issue to help people find the bug / config error. Attaching run logs is always super helpful too.

I always recommend using as little plugins as possible, as there is very little testing of applying them all together. Interoperability of all the filtering plugins has never been super tested and the apply order isn't documented / tested.

I feel here cause you have on setuptools in allowlist is the only package you pull?

[joeboughazali]

so I removed all plugins and kept the following as you suggested

[mirror]
directory = /data/MINI_PYPI/bandersnatch/packages/
master = https://pypi.org
timeout = 20
workers = 3
hash-index = false
stop-on-error = false
json = true

[plugins]
enabled =
        allowlist_project
        blocklist_project
        exclude_platform

[allowlist]
packages =
        setuptools

[blocklist]
platforms =
        windows
        macos
        freebsd
        py3
        py3.0
        py3.1
        py3.2
        py3.3
        py3.4
        py3.5
        py3.6
        py3.7
        py3.8
        py3.9
        py3.10
        py3.11

I thought this would block the sync of all python3-compatible versions of setuptools. But it pulled all versions. This is the log:

2023-07-12 09:22:55,420 INFO: No status file to move (/data/MINI_PYPI/bandersnatch/packages/status) - Full sync will occur (main.py:187)
2023-07-12 09:22:55,420 INFO: Selected storage backend: filesystem (configuration.py:131)
2023-07-12 09:22:55,420 INFO: Selected compare method: hash (configuration.py:179)
2023-07-12 09:22:55,431 INFO: Initialized project plugin allowlist_project, filtering ['setuptools'] (allowlist_name.py:33)
2023-07-12 09:22:55,432 INFO: Initialized project plugin blocklist_project, filtering [] (blocklist_name.py:27)
2023-07-12 09:22:55,446 INFO: Initialized exclude_platform plugin with ['.win32', '-win32', 'win_amd64', 'win-amd64', 'macosx_', 'macosx-', '.freebsd', '-freebsd', '-cp30-', '-pp30-', '-ip30-', '-jy30-', '-py3.0-', '-py3.0.', '-cp31-', '-pp31-', '-ip31-', '-jy31-', '-py3.1-', '-py3.1.', '-cp32-', '-pp32-', '-ip32-', '-jy32-', '-py3.2-', '-py3.2.', '-cp33-', '-pp33-', '-ip33-', '-jy33-', '-py3.3-', '-py3.3.', '-cp34-', '-pp34-', '-ip34-', '-jy34-', '-py3.4-', '-py3.4.', '-cp35-', '-pp35-', '-ip35-', '-jy35-', '-py3.5-', '-py3.5.', '-cp36-', '-pp36-', '-ip36-', '-jy36-', '-py3.6-', '-py3.6.', '-cp37-', '-pp37-', '-ip37-', '-jy37-', '-py3.7-', '-py3.7.', '-cp38-', '-pp38-', '-ip38-', '-jy38-', '-py3.8-', '-py3.8.', '-cp39-', '-pp39-', '-ip39-', '-jy39-', '-py3.9-', '-py3.9.', '-cp310-', '-pp310-', '-ip310-', '-jy310-', '-py3.10-', '-py3.10.'] (filename_name.py:108)
2023-07-12 09:22:55,449 INFO: Status file /data/MINI_PYPI/bandersnatch/packages/status missing. Starting over. (mirror.py:566)
2023-07-12 09:22:55,449 INFO: Syncing with https://pypi.org. (mirror.py:57)
2023-07-12 09:22:55,449 INFO: Current mirror serial: 0 (mirror.py:278)
2023-07-12 09:22:55,449 INFO: Syncing all packages. (mirror.py:293)
2023-07-12 09:23:10,527 INFO: Package 'setuptools' is allowlisted (allowlist_name.py:90)
2023-07-12 09:23:10,735 INFO: Trying to reach serial: 18856834 (mirror.py:310)
2023-07-12 09:23:10,735 INFO: 1 packages to sync. (mirror.py:312)
2023-07-12 09:23:10,745 INFO: No metadata filters are enabled. Skipping metadata filtering (mirror.py:76)
2023-07-12 09:23:10,746 INFO: No release filters are enabled. Skipping release filtering (mirror.py:78)
2023-07-12 09:23:10,746 INFO: Fetching metadata for package: setuptools (serial 18561969) (package.py:58)

To answer your question, for now, I only need setup-tools. My aim is to sync only needed packages. I don't want all of the packages present in my repo

[cooperlees]

Thanks for the logs etc. - But in future please add --debug to get more logging. So I was able to repro locally with your condif:

python3 -m venv --upgrade-deps /tmp/tb
/tmp/tb/bin/pip install -e .
vim /tmp/bandersnatch.conf
mkdir /tmp/pypi
/tmp/tb/bin/bandersnatch -c /tmp/bandersnatch.conf --debug mirror

Can confirm reading the code that all sdists are expected. But the -py3- wheels are not. So need to workout if the bug is:

1. Caused by the allowlist being used with denylist causing the latter to be ignored
2. Is there a bug with the release filename plugin itself
   • It does have tests tho so we could maybe use them, to test this theory

All help here is welcome. PRs welcome. I am down on time ...

[cooperlees]

I believe I have fixed the bug. That said, you might want to just use version filtering for package in an allow list to just pull version that support python 2 as this method is still going to pull down all sdists (source distributions) of all new versions for the allow listed package.

For example:

[plugins]
enabled =
    allowlist_release

[allowlist]
packages =
    setuptools<=45.0.0

• I believe 45.0.0 was the last python 2 supported version of setuptools

https://bandersnatch.readthedocs.io/en/latest/filtering_configuration.html#packages

(I haven't tested this tho)

[cooperlees]

I lie - CI does use this feature: https://github.com/pypa/bandersnatch/blob/main/src/bandersnatch/tests/ci.conf#L26