bandersnatch verify key error 'version'

[jax79sg]

Version: Bandersnatch docker image (pulled at time of this message ). Issues: When performing verify

• It doesn't seem to consider whitelist_project plugin. E.g. Non whitelisted items were downloaded as well
• It fails with the latest_release plugin with error for some packages.

  2020-08-15 10:40:37,867 INFO: Parsing cherrypy
  2020-08-15 10:40:37,875 INFO: Initialized latest releases plugin with keep=80
  Traceback (most recent call last):
  File "/usr/local/bin/bandersnatch", line 33, in <module>
  sys.exit(load_entry_point('bandersnatch==4.1.1', 'console_scripts', 'bandersnatch')())
  File "/usr/local/lib/python3.8/site-packages/bandersnatch/main.py", line 203, in main
  return loop.run_until_complete(async_main(args, config))
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
  return future.result()
  File "/usr/local/lib/python3.8/site-packages/bandersnatch/main.py", line 115, in async_main
  return await bandersnatch.verify.metadata_verify(config, args)
  File "/usr/local/lib/python3.8/site-packages/bandersnatch/verify.py", line 210, in metadata_verify
  await verify_producer(
  File "/usr/local/lib/python3.8/site-packages/bandersnatch/verify.py", line 173, in verify_producer
  await asyncio.gather(
  File "/usr/local/lib/python3.8/site-packages/bandersnatch/verify.py", line 163, in consume
  await verify(
  File "/usr/local/lib/python3.8/site-packages/bandersnatch/verify.py", line 119, in verify
  plugin.filter(pkg)
  File "/usr/local/lib/python3.8/site-packages/bandersnatch_filter_plugins/latest_name.py", line 67, in filter
  version = metadata["version"]
  KeyError: 'version'

  Here is my bandersnatch.conf for reference

  
  [mirror]
  directory = /home/jax/bigwhite/pypi

; Save JSON metadata into the web tree: ; URL/pypi/PKG_NAME/json (Symlink) -> URL/json/PKG_NAME json = true

; Cleanup legacy non PEP 503 normalized named simple directories cleanup = false

; The PyPI server which will be mirrored. ; master = https://test.python.org ; scheme for PyPI server MUST be https master = https://pypi.org

; The network socket timeout to use for all connections. This is set to a ; somewhat aggressively low value: rather fail quickly temporarily and re-run ; the client soon instead of having a process hang infinitely and have TCP not ; catching up for ages. timeout = 60000

; Number of worker threads to use for parallel downloads. ; Recommendations for worker thread setting: ; - leave the default of 3 to avoid overloading the pypi master ; - official servers located in data centers could run 10 workers ; - anything beyond 10 is probably unreasonable and avoided by bandersnatch workers = 3

; Whether to hash package indexes ; Note that package index directory hashing is incompatible with pip, and so ; this should only be used in an environment where it is behind an application ; that can translate URIs to filesystem locations. For example, with the ; following Apache RewriteRule: ; RewriteRule ^([^/])([^/])/$ /mirror/pypi/web/simple/$1/$1$2/ ; RewriteRule ^([^/])([^/])/([^/]+)$/ /mirror/pypi/web/simple/$1/$1$2/$3 ; OR ; following nginx rewrite rules: ; rewrite ^/simple/([^/])([^/])/$ /simple/$1/$1$2/ last; ; rewrite ^/simple/([^/])([^/])/([^/]+)$/ /simple/$1/$1$2/$3 last; ; Setting this to true would put the package 'abc' index in simple/a/abc. ; Recommended setting: the default of false for full pip/pypi compatibility. hash-index = false

; Whether to stop a sync quickly after an error is found or whether to continue ; syncing but not marking the sync as successful. Value should be "true" or ; "false". stop-on-error = false

; Advanced logging configuration. Uncomment and set to the location of a ; python logging format logging config file. ; log-config = /etc/bandersnatch-log.conf

; Generate index pages with absolute urls rather than relative links. This is ; generally not necessary, but was added for the official internal PyPI mirror, ; which requires serving packages from https://files.pythonhosted.org ; root_uri = https://example.com

; Number of consumers which verify metadata verifiers = 3

; Number of prior simple index.html to store. Used as a safeguard against ; upstream changes generating blank index.html files. Prior versions are ; stored under as "versions/index.html" and the current ; index.html will be a symlink to the latest version. ; If set to 0 no prior versions are stored and index.html is the latest version. ; If unset defaults to 0. ; keep_index_versions = 0

; vim: set ft=cfg:

; Configure a file to write out the list of files downloaded during the mirror. ; This is useful for situations when mirroring to offline systems where a process ; is required to only sync new files to the upstream mirror. ; The file be be named as set in the diff-file, and overwritten unless the ; diff-append-epoch setting is set to true. If this is true, the epoch date will ; be appended to the filename (i.e. /path/to/diff-1568129735) ; diff-file = /srv/pypi/mirrored-files ; diff-append-epoch = true

[plugins] enabled = exclude_platform whitelist_project latest_release

[blacklist] platforms = windows macos freebsd

[whitelist] packages = urllib3 ....truncated [latest_release] keep = 80

[cooperlees]

verify does not respect plugins at all. It never has. Only mirror. Would accept a PR that fixes that. Maybe this need to be clearer in the documentation until we do integrate support (if we ever do).

The key error is a bug that definitely needs to be fixed. Thanks for reporting.

[jax79sg]

Hi @cooperlees , thank you for the reply. Due to my poor internet connection, I get disconnections often and thus result in incomplete mirror. Other than verify, what can I do to download the missing delta?

[cooperlees]

bandersnatch sync might help you to fix a specific package.

cooper-mbp1:bandersnatch cooper$ /tmp/bla_venv/bin/bandersnatch sync --help
usage: bandersnatch sync [-h] package [package ...]

positional arguments:
  package     The name of package to sync

[cooperlees]

This error may have been fixed with some plugin work. If not, please re-open.

[xoroz]

The error still exists.. I am using a few plugin and when using verify command I see

Traceback (most recent call last): File "/opt/repos/PyPI/bandersnatch/bin/bandersnatch", line 8, in sys.exit(main()) File "/opt/repos/PyPI/bandersnatch/lib64/python3.8/site-packages/bandersnatch/main.py", line 219, in main return asyncio.run(async_main(args, config)) File "/usr/lib64/python3.8/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/opt/repos/PyPI/bandersnatch/lib64/python3.8/site-packages/bandersnatch/main.py", line 157, in async_main return await bandersnatch.verify.metadata_verify(config, args) File "/opt/repos/PyPI/bandersnatch/lib64/python3.8/site-packages/bandersnatch/verify.py", line 259, in metadata_verify await verify_producer( File "/opt/repos/PyPI/bandersnatch/lib64/python3.8/site-packages/bandersnatch/verify.py", line 222, in verify_producer await asyncio.gather( File "/opt/repos/PyPI/bandersnatch/lib64/python3.8/site-packages/bandersnatch/verify.py", line 212, in consume await verify( File "/opt/repos/PyPI/bandersnatch/lib64/python3.8/site-packages/bandersnatch/verify.py", line 155, in verify plugin.filter(pkg) File "/opt/repos/PyPI/bandersnatch/lib64/python3.8/site-packages/bandersnatch_filter_plugins/latest_name.py", line 43, in filter version: str = metadata["version"] KeyError: 'version'

[plugins] enabled = exclude_platform latest_release blocklist_project blocklist_release allowlist_project allowlist_release project_requirements project_requirements_pinned