search

damnever / pigar

:coffee: A tool to generate requirements.txt for Python project, and more than that. (IT IS NOT A PACKAGE MANAGEMENT TOOL)
https://damnever.github.io/pigar/
BSD 3-Clause "New" or "Revised" License
1.61k stars 90 forks source link

pigar -c requirements.txt download error if lib name contains hyphen or brackets (- or []) #113

Closed mickdewald closed 1 year ago

mickdewald commented 1 year ago

Problems for downloading info if lib name contains hypen (- like python-multipart) or brackets ([], like uvicorn[standard])

If I try to list information for libraries like this:

./venv/bin/pigar -c requirements.txt I run into some errors:

❯ ./venv/bin/pigar -c requirements.txt --no
--- Logging error ---
Traceback (most recent call last):
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/core.py", line 248, in check_requirements_latest_versions
    latest = downloader.download_package(pkg).version()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/pypi.py", line 64, in download_package
    pkg_info = self.download(PKG_INFO_URL.format(name))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/pypi.py", line 54, in download
    resp.raise_for_status()
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/pypi/uvicorn%5Bstandard%5D/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/logging/__init__.py", line 1110, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 953, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/log.py", line 49, in format
    message = color(record.getMessage())
                    ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 377, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
TypeError: must be real number, not HTTPError
Call stack:
  File "/home/mick/projects/capmon-backend/./venv/bin/pigar", line 8, in <module>
    sys.exit(main())
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/__main__.py", line 174, in main
    check_requirements_latest_versions(
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/core.py", line 250, in check_requirements_latest_versions
    logger.error('checking %s failed: %e', pkg, e)
Message: 'checking %s failed: %e'
Arguments: ('uvicorn[standard]', HTTPError('404 Client Error: Not Found for url: https://pypi.org/pypi/uvicorn%5Bstandard%5D/json'))
--- Logging error ---
Traceback (most recent call last):
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/core.py", line 248, in check_requirements_latest_versions
    latest = downloader.download_package(pkg).version()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/pypi.py", line 64, in download_package
    pkg_info = self.download(PKG_INFO_URL.format(name))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/pypi.py", line 54, in download
    resp.raise_for_status()
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/pypi/python-jose%5Bcryptography%5D/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/logging/__init__.py", line 1110, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 953, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/log.py", line 49, in format
    message = color(record.getMessage())
                    ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 377, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
TypeError: must be real number, not HTTPError
Call stack:
  File "/home/mick/projects/capmon-backend/./venv/bin/pigar", line 8, in <module>
    sys.exit(main())
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/__main__.py", line 174, in main
    check_requirements_latest_versions(
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/core.py", line 250, in check_requirements_latest_versions
    logger.error('checking %s failed: %e', pkg, e)
Message: 'checking %s failed: %e'
Arguments: ('python-jose[cryptography]', HTTPError('404 Client Error: Not Found for url: https://pypi.org/pypi/python-jose%5Bcryptography%5D/json'))
--- Logging error ---
Traceback (most recent call last):
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/core.py", line 248, in check_requirements_latest_versions
    latest = downloader.download_package(pkg).version()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/pypi.py", line 64, in download_package
    pkg_info = self.download(PKG_INFO_URL.format(name))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/pypi.py", line 54, in download
    resp.raise_for_status()
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/pypi/passlib%5Bbcrypt%5D/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/logging/__init__.py", line 1110, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 953, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/log.py", line 49, in format
    message = color(record.getMessage())
                    ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/__init__.py", line 377, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
TypeError: must be real number, not HTTPError
Call stack:
  File "/home/mick/projects/capmon-backend/./venv/bin/pigar", line 8, in <module>
    sys.exit(main())
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/__main__.py", line 174, in main
    check_requirements_latest_versions(
  File "/home/mick/projects/capmon-backend/venv/lib/python3.11/site-packages/pigar/core.py", line 250, in check_requirements_latest_versions
    logger.error('checking %s failed: %e', pkg, e)
Message: 'checking %s failed: %e'
Arguments: ('passlib[bcrypt]', HTTPError('404 Client Error: Not Found for url: https://pypi.org/pypi/passlib%5Bbcrypt%5D/json'))
 ==============================================
  PACKAGE                   | CURRENT | LATEST
  --------------------------+---------+-------
  fastapi                   | 0.85.0  | 0.85.1
  h11                       | 0.12.0  | 0.14.0
  pydantic                  | 1.10.2  | 1.10.2
  starlette                 | 0.20.4  | 0.21.0
  uvicorn[standard]         |         | 0.21.0
  python-socketio           |         | 5.7.2
  websockets                | 10.4    | 10.4
  aiohttp                   | 3.8.3   | 3.8.3
  httpx                     | 0.23.0  | 0.23.0
  python-jose[cryptography] |         | 0.23.0
  passlib[bcrypt]           |         | 0.23.0
  bcrypt                    | 4.0.1   | 4.0.1
  yahooquery                | 2.2.15  | 2.2.15
  yfinance                  | 0.1.84  | 0.1.84
  html5lib                  | 1.1     | 1.1
  pycoingecko               | 3.0.0   | 3.0.0
  beautifulsoup4            | 4.11.1  | 4.11.1
  cloudscraper              | 1.2.64  | 1.2.64
  motor                     | 3.1.1   | 3.1.1
  pymongo                   | 4.3.2   | 4.3.2
  beanie                    | 1.13.0  | 1.13.0
  kubernetes                | 25.3.0  | 25.3.0
  pytest                    | 7.2.0   | 7.2.0
  pytest-asyncio            |         | 0.20.1
  locust                    | 2.12.2  | 2.12.2
  loguru                    | 0.6.0   | 0.6.0
  autopep8                  | 1.7.0   | 1.7.0
  pylint                    | 2.15.5  | 2.15.5
  python-multipart          |         | 0.0.5
  email-validator           |         | 1.3.0
  configparser              | 5.3.0   | 5.3.0
  apscheduler               |         | 3.9.1
  Pillow                    | 9.2.0   | 9.2.0
  async_lru                 | 1.0.3   | 1.0.3
  colorama                  | 0.4.6   | 0.4.6
  pigar                     | 1.0.0   | 1.0.0
 ==============================================
damnever commented 1 year ago

Try the main branch pip install git+https://github.com/damnever/pigar.git@main --upgrade

mickdewald commented 1 year ago

Thanks for answering. I will test this a soon I have access to my PC. Is there a way to make this work in requirements.txt? Like pigar.git@main?

EDIT I think I found a good solution here https://stackoverflow.com/a/35998253/7220665

I use this in my project to print out the libs including versions as some debug information


# https://capmon.org/api/server

{"message":"Welcome to the capmon-server v86bfabe1_2022-11-05_19:04:34","python":"3.11.0","libs":
{"fastapi":{"current":"0.85.1","latest":"0.86.0"},
"h11":{"current":"0.12.0","latest":"0.14.0"},
"pydantic":{"current":"1.10.2","latest":"1.10.2"},
...
mickdewald commented 1 year ago

I tested your suggested solution, and here is my result:

  PACKAGE          | SPEC     | LOCAL  | LATEST
  -----------------+----------+--------+-------
  fastapi          | ==0.85.1 | 0.85.1 | 0.86.0
  h11              | ==0.12.0 | 0.12.0 | 0.14.0
  pydantic         |          | 1.10.2 | 1.10.2
  starlette        | ==0.20.4 | 0.20.4 | 0.21.0
  uvicorn          |          | 0.19.0 | 0.19.0
  python-socketio  |          |        | 5.7.2
  websockets       |          | 10.4   | 10.4
  aiohttp          |          | 3.8.3  | 3.8.3
  httpx            |          | 0.23.0 | 0.23.0
  python-jose      |          |        | 3.3.0
  passlib          |          | 1.7.4  | 1.7.4
  bcrypt           |          | 4.0.1  | 4.0.1
  yahooquery       |          | 2.2.15 | 2.2.15
  yfinance         |          | 0.1.85 | 0.1.85
  html5lib         |          | 1.1    | 1.1
  pycoingecko      |          | 3.1.0  | 3.1.0
  beautifulsoup4   |          | 4.11.1 | 4.11.1
  cloudscraper     |          | 1.2.64 | 1.2.64
  motor            |          | 3.1.1  | 3.1.1
  pymongo          |          | 4.3.2  | 4.3.2
  beanie           |          | 1.15.1 | 1.15.1
  kubernetes       |          | 25.3.0 | 25.3.0
  pytest           |          | 7.2.0  | 7.2.0
  pytest-asyncio   |          |        | 0.20.1
  locust           |          | 2.13.0 | 2.13.0
  loguru           |          | 0.6.0  | 0.6.0
  autopep8         |          | 2.0.0  | 2.0.0
  pylint           |          | 2.15.5 | 2.15.5
  python-multipart |          |        | 0.0.5
  email-validator  |          |        | 1.3.0
  configparser     |          | 5.3.0  | 5.3.0
  apscheduler      |          |        | 3.9.1
  Pillow           |          | 9.3.0  | 9.3.0
  async_lru        |          | 1.0.3  | 1.0.3
  colorama         |          | 0.4.6  | 0.4.6
 ===============================================

The lib definitions including brackets seem to be working now, but the ones with hyphens do not show a version number and there are still exceptions:

--- Logging error ---
Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/packaging/requirements.py", line 102, in __init__
    req = REQUIREMENT.parseString(requirement_string)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/pyparsing/core.py", line 1141, in parse_string
    raise exc.with_traceback(None)
pyparsing.exceptions.ParseException: Expected string_end, found '+'  (at char 3), (line:1, col:4)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/pigar/helpers.py", line 131, in parse_requirements
    req = Requirement(line)
          ^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/packaging/requirements.py", line 104, in __init__
    raise InvalidRequirement(
packaging.requirements.InvalidRequirement: Parse error at "'+https:/'": Expected string_end

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/pigar/core.py", line 241, in check_requirements_latest_versions
    for req in parse_requirements(file):
  File "/root/.local/lib/python3.11/site-packages/pigar/helpers.py", line 133, in parse_requirements
    raise PraseRequirementError('line {}: {}'.format(lineno, e))
pigar.helpers.PraseRequirementError: line 226: Parse error at "'+https:/'": Expected string_end

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/logging/__init__.py", line 1110, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/logging/__init__.py", line 953, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/pigar/log.py", line 49, in format
    message = color(record.getMessage())
                    ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/logging/__init__.py", line 377, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
TypeError: must be real number, not PraseRequirementError
Call stack:
  File "/root/.local/bin/pigar", line 8, in <module>
    sys.exit(main())
  File "/root/.local/lib/python3.11/site-packages/pigar/__main__.py", line 174, in main
    check_requirements_latest_versions(
  File "/root/.local/lib/python3.11/site-packages/pigar/core.py", line 256, in check_requirements_latest_versions
    logger.error('parse %s failed: %e', file, e)
Message: 'parse %s failed: %e'
Arguments: ('/server/requirements.txt', PraseRequirementError('line 226: Parse error at "\'+https:/\'": Expected string_end'))
damnever commented 1 year ago

Looks like there are some invalid lines in requirements.txt, could you post your requirements.txt file here?

mickdewald commented 1 year ago

So here is the requirements.txt:

fastapi==0.86.0

h11==0.12.0

pydantic==1.10.2

starlette==0.20.4

uvicorn[standard]==0.19

python-socketio==5.7.2

websockets==10.4

aiohttp==3.8.3

httpx==0.23.0

python-jose[cryptography]==3.3.0

passlib[bcrypt]==1.7.4

bcrypt==4.0.1

yahooquery==2.2.15

yfinance==0.1.84

html5lib==1.1

pycoingecko==3.1.0

beautifulsoup4==4.11.1

cloudscraper==1.2.64

motor==3.1.1

pymongo==4.3.2

beanie==1.15.1

kubernetes==25.3.0

pytest==7.2.0

pytest-asyncio==0.20.1

locust==2.12.0

loguru==0.6.0

autopep8==2.0.0

pylint==2.15.5

python-multipart==0.0.5

email-validator==1.3.0

configparser==5.3.0

apscheduler==3.9.1

Pillow==9.3.0

async_lru==1.0.3

colorama==0.4.6

git+https://github.com/damnever/pigar.git@main

And this is the error message:

--- Logging error ---
Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/packaging/requirements.py", line 102, in __init__
    req = REQUIREMENT.parseString(requirement_string)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/pyparsing/core.py", line 1141, in parse_string
    raise exc.with_traceback(None)
pyparsing.exceptions.ParseException: Expected string_end, found '+'  (at char 3), (line:1, col:4)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/pigar/helpers.py", line 131, in parse_requirements
    req = Requirement(line)
          ^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/packaging/requirements.py", line 104, in __init__
    raise InvalidRequirement(
packaging.requirements.InvalidRequirement: Parse error at "'+https:/'": Expected string_end

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/pigar/core.py", line 241, in check_requirements_latest_versions
    for req in parse_requirements(file):
  File "/root/.local/lib/python3.11/site-packages/pigar/helpers.py", line 133, in parse_requirements
    raise PraseRequirementError('line {}: {}'.format(lineno, e))
pigar.helpers.PraseRequirementError: line 70: Parse error at "'+https:/'": Expected string_end

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/logging/__init__.py", line 1110, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/logging/__init__.py", line 953, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/pigar/log.py", line 49, in format
    message = color(record.getMessage())
                    ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/logging/__init__.py", line 377, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
TypeError: must be real number, not PraseRequirementError
Call stack:
  File "/root/.local/bin/pigar", line 8, in <module>
    sys.exit(main())
  File "/root/.local/lib/python3.11/site-packages/pigar/__main__.py", line 174, in main
    check_requirements_latest_versions(
  File "/root/.local/lib/python3.11/site-packages/pigar/core.py", line 256, in check_requirements_latest_versions
    logger.error('parse %s failed: %e', file, e)
Message: 'parse %s failed: %e'
Arguments: ('/server/requirements.txt', PraseRequirementError('line 70: Parse error at "\'+https:/\'": Expected string_end'))

But nevertheless, there is an output:

  PACKAGE          | SPEC     | LOCAL  | LATEST
  -----------------+----------+--------+-------
  fastapi          | ==0.86.0 | 0.86.0 | 0.86.0
  h11              | ==0.12.0 | 0.12.0 | 0.14.0
  pydantic         | ==1.10.2 | 1.10.2 | 1.10.2
  starlette        | ==0.20.4 | 0.20.4 | 0.21.0
  uvicorn          | ==0.19   | 0.19.0 | 0.19.0
  python-socketio  | ==5.7.2  |        | 5.7.2
  websockets       | ==10.4   | 10.4   | 10.4
  aiohttp          | ==3.8.3  | 3.8.3  | 3.8.3
  httpx            | ==0.23.0 | 0.23.0 | 0.23.0
  python-jose      | ==3.3.0  |        | 3.3.0
  passlib          | ==1.7.4  | 1.7.4  | 1.7.4
  bcrypt           | ==4.0.1  | 4.0.1  | 4.0.1
  yahooquery       | ==2.2.15 | 2.2.15 | 2.2.15
  yfinance         | ==0.1.84 | 0.1.84 | 0.1.85
  html5lib         | ==1.1    | 1.1    | 1.1
  pycoingecko      | ==3.1.0  | 3.1.0  | 3.1.0
  beautifulsoup4   | ==4.11.1 | 4.11.1 | 4.11.1
  cloudscraper     | ==1.2.64 | 1.2.64 | 1.2.64
  motor            | ==3.1.1  | 3.1.1  | 3.1.1
  pymongo          | ==4.3.2  | 4.3.2  | 4.3.2
  beanie           | ==1.15.1 | 1.15.1 | 1.15.1
  kubernetes       | ==25.3.0 | 25.3.0 | 25.3.0
  pytest           | ==7.2.0  | 7.2.0  | 7.2.0
  pytest-asyncio   | ==0.20.1 |        | 0.20.1
  locust           | ==2.12.0 | 2.12.0 | 2.13.0
  loguru           | ==0.6.0  | 0.6.0  | 0.6.0
  autopep8         | ==2.0.0  | 2.0.0  | 2.0.0
  pylint           | ==2.15.5 | 2.15.5 | 2.15.5
  python-multipart | ==0.0.5  |        | 0.0.5
  email-validator  | ==1.3.0  |        | 1.3.0
  configparser     | ==5.3.0  | 5.3.0  | 5.3.0
  apscheduler      | ==3.9.1  |        | 3.9.1
  Pillow           | ==9.3.0  | 9.3.0  | 9.3.0
  async_lru        | ==1.0.3  | 1.0.3  | 1.0.3
  colorama         | ==0.4.6  | 0.4.6  | 0.4.6
 ===============================================

Maybe the reason for the error message is the pigar requirement itself, since it's not part of the table?

damnever commented 1 year ago

You can remove git+https://github.com/damnever/pigar.git@main to fix the issue for now..

In the case like git+https://github.com/damnever/pigar.git@main, the first step may be to download the package, then to parse information from it, there is much work to be done and done it right.

mickdewald commented 1 year ago

You can remove git+https://github.com/damnever/pigar.git@main to fix the issue for now..

I just use the old version for now and live with the error message. It's just for debugging, so that's fine right now

In the case like git+https://github.com/damnever/pigar.git@main, the first step may be to download the package, then to parse information from it, there is much work to be done and done it right.

I am looking forward to upcoming versions. Thanks for this great tool!