search

pypa / bandersnatch

A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/
Academic Free License v3.0
448 stars 141 forks source link

bandersnatch verify --delete occurs error #1488

Open 89ao opened 1 year ago

89ao commented 1 year ago

sh-4.2# bandersnatch --version bandersnatch 6.3.0

sh-4.2# bandersnatch verify --delete --json-update
2023-07-06 16:13:58,112 bandersnatch.verify: INFO Starting verify for /repo with 10 workers
2023-07-06 16:13:59,897 bandersnatch.verify: INFO Parsing 101903762
2023-07-06 16:13:59,897 bandersnatch.master: INFO Fetching https://pypi.org/pypi/101903762/json
2023-07-06 16:14:00,220 bandersnatch: INFO Initialized release plugin blocklist_release, filtering [<Requirement('tensorflow-io-nightly')>, <Requirement('tf-nightly-cpu')>, <Requirement('tf-nightly')>, <Requirement('appium')>, <Requirement('pyagrum-nightly')>, <Requirement('tf-nightly-gpu')>, <Requirement('uselesscapitalquiz')>]
Traceback (most recent call last):
  File "/usr/local/bin/bandersnatch", line 8, in <module>
    sys.exit(main())
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch/main.py", line 226, in main
    return asyncio.run(async_main(args, config))
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch/main.py", line 162, in async_main
    return await bandersnatch.verify.metadata_verify(config, args)
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch/verify.py", line 269, in metadata_verify
    await verify_producer(
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch/verify.py", line 227, in verify_producer
    await asyncio.gather(
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch/verify.py", line 217, in consume
    await verify(
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch/verify.py", line 160, in verify
    plugin.filter(pkg)
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch_filter_plugins/blocklist_name.py", line 146, in filter
    version = metadata["version"]
KeyError: 'version'
cooperlees commented 1 year ago

Interesting, must be a package with no versions. Can you capture the --debug output and attach please and I'll see what the package metadata looks like and make the code handle this metadata state.

Mainly I just want to see the JSON for the package it's failing on to see how to handle it correctly. Should be an easy fix.

89ao commented 1 year ago

the debug log is below:

sh-4.2# bandersnatch --debug verify --delete
2023-07-06 18:57:04,396 bandersnatch.verify: INFO Starting verify for /repo with 10 workers
2023-07-06 18:57:14,176 bandersnatch.verify: INFO Parsing 101903762
2023-07-06 18:57:14,213 bandersnatch: INFO Initialized release plugin blocklist_release, filtering [<Requirement('tf-nightly-gpu')>, <Requirement('tf-nightly-cpu')>, <Requirement('pyagrum-nightly')>, <Requirement('uselesscapitalquiz')>, <Requirement('tensorflow-io-nightly')>, <Requirement('tf-nightly')>, <Requirement('appium')>]
Traceback (most recent call last):
  File "/usr/local/bin/bandersnatch", line 8, in <module>
    sys.exit(main())
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch/main.py", line 226, in main
    return asyncio.run(async_main(args, config))
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch/main.py", line 162, in async_main
    return await bandersnatch.verify.metadata_verify(config, args)
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch/verify.py", line 269, in metadata_verify
    await verify_producer(
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch/verify.py", line 227, in verify_producer
    await asyncio.gather(
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch/verify.py", line 217, in consume
    await verify(
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch/verify.py", line 160, in verify
    plugin.filter(pkg)
  File "/data/bandersnatch/lib/python3.9/site-packages/bandersnatch_filter_plugins/blocklist_name.py", line 146, in filter
    version = metadata["version"]
KeyError: 'version'
sh-4.2# cat /repo/web/simple/101903762/index.html
<!DOCTYPE html>
<html>
  <head>
    <meta name="pypi:repository-version" content="1.0">
    <title>Links for 101903762</title>
  </head>
  <body>
    <h1>Links for 101903762</h1>
    <a href="../../packages/0a/29/8ee8281a7d9f43c544e5bc0b2433449a40d838dca1a444385d54422aa2a0/101903762-0.0.1-py3-none-any.whl#sha256=9ac8b5f28832285c63239412a522d8543d380829323a3a02a66df3a7a3cff346">101903762-0.0.1-py3-none-any.whl</a><br/>
    <a href="../../packages/e1/f4/269740bd163001bcab48975f49b3a0eec70a6919ffdfaea28f74ddb9d7d3/101903762-0.0.1.tar.gz#sha256=07c81e62d19da21e2fd970dfdceec02ddeaed56a746135bf0cfafc7259c0dc32">101903762-0.0.1.tar.gz</a><br/>
  </body>
</html>
<!--SERIAL 13026825-->

the package "101903762"'s json is like this:

sh-4.2# cat /repo/web/pypi/101903762/json
{
    "info": {
        "author": "Divyanshu",
        "author_email": "<djindal1_be19@thapar.edu>",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 1 - Planning",
            "Intended Audience :: Developers",
            "Operating System :: MacOS :: MacOS X",
            "Operating System :: Microsoft :: Windows",
            "Operating System :: Unix",
            "Programming Language :: Python :: 3"
        ],
        "description": "",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "",
        "keywords": "python,video,stream,video stream,camera stream,sockets",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "101903762",
        "package_url": "https://pypi.org/project/101903762/",
        "platform": "",
        "project_url": "https://pypi.org/project/101903762/",
        "project_urls": null,
        "release_url": "https://pypi.org/project/101903762/0.0.1/",
        "requires_dist": [
            "pandas",
            "numpy"
        ],
        "requires_python": "",
        "summary": "Calculating topsis Score",
        "version": "0.0.1",
        "yanked": false,
        "yanked_reason": null
    },
    "last_serial": 13026825,
    "releases": {
        "0.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "blake2b_256": "0a298ee8281a7d9f43c544e5bc0b2433449a40d838dca1a444385d54422aa2a0",
                    "md5": "97645b3daf7a93e6b436848780577e49",
                    "sha256": "9ac8b5f28832285c63239412a522d8543d380829323a3a02a66df3a7a3cff346"
                },
                "downloads": -1,
                "filename": "101903762-0.0.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "97645b3daf7a93e6b436848780577e49",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 2272,
                "upload_time": "2022-02-27T18:38:06",
                "upload_time_iso_8601": "2022-02-27T18:38:06.139267Z",
                "url": "https://files.pythonhosted.org/packages/0a/29/8ee8281a7d9f43c544e5bc0b2433449a40d838dca1a444385d54422aa2a0/101903762-0.0.1-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "blake2b_256": "e1f4269740bd163001bcab48975f49b3a0eec70a6919ffdfaea28f74ddb9d7d3",
                    "md5": "162d6e4f5ad0df4546450e94d39a569a",
                    "sha256": "07c81e62d19da21e2fd970dfdceec02ddeaed56a746135bf0cfafc7259c0dc32"
                },
                "downloads": -1,
                "filename": "101903762-0.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "162d6e4f5ad0df4546450e94d39a569a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2058,
                "upload_time": "2022-02-27T18:38:08",
                "upload_time_iso_8601": "2022-02-27T18:38:08.263032Z",
                "url": "https://files.pythonhosted.org/packages/e1/f4/269740bd163001bcab48975f49b3a0eec70a6919ffdfaea28f74ddb9d7d3/101903762-0.0.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "0a298ee8281a7d9f43c544e5bc0b2433449a40d838dca1a444385d54422aa2a0",
                "md5": "97645b3daf7a93e6b436848780577e49",
                "sha256": "9ac8b5f28832285c63239412a522d8543d380829323a3a02a66df3a7a3cff346"
            },
            "downloads": -1,
            "filename": "101903762-0.0.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "97645b3daf7a93e6b436848780577e49",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 2272,
            "upload_time": "2022-02-27T18:38:06",
            "upload_time_iso_8601": "2022-02-27T18:38:06.139267Z",
            "url": "https://files.pythonhosted.org/packages/0a/29/8ee8281a7d9f43c544e5bc0b2433449a40d838dca1a444385d54422aa2a0/101903762-0.0.1-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "e1f4269740bd163001bcab48975f49b3a0eec70a6919ffdfaea28f74ddb9d7d3",
                "md5": "162d6e4f5ad0df4546450e94d39a569a",
                "sha256": "07c81e62d19da21e2fd970dfdceec02ddeaed56a746135bf0cfafc7259c0dc32"
            },
            "downloads": -1,
            "filename": "101903762-0.0.1.tar.gz",
            "has_sig": false,
            "md5_digest": "162d6e4f5ad0df4546450e94d39a569a",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 2058,
            "upload_time": "2022-02-27T18:38:08",
            "upload_time_iso_8601": "2022-02-27T18:38:08.263032Z",
            "url": "https://files.pythonhosted.org/packages/e1/f4/269740bd163001bcab48975f49b3a0eec70a6919ffdfaea28f74ddb9d7d3/101903762-0.0.1.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "vulnerabilities": []
}