Closed TuriJ95 closed 1 year ago
This is an issue for Poetry-specified dependencies too - packages from [tool.poetry.group.dev.dependencies]
are now being included by licensecheck
by default.
Yes, it seems that are automatically parsed all the optional dependencies (a.k.a. "extras") of every listed dependency, whether or not they are actually tracked in the requirements section. Probably lines 111-126 of get_deps.py should be reworked.
Actually they are:
requirementsWithDeps = reqs.copy()
for requirement in reqs:
try:
pkgMetadata = metadata.metadata(requirement)
for req in [resolveReq(req) for req in pkgMetadata.get_all("Requires-Dist") or []]:
requirementsWithDeps.add(req)
except metadata.PackageNotFoundError:
request = session.get(f"https://pypi.org/pypi/{requirement}/json", timeout=60)
response = request.json()
try:
for req in [resolveReq(req) for req in response["info"]["requires_dist"]]:
requirementsWithDeps.add(req)
except (KeyError, TypeError):
pass
return requirementsWithDeps
and should be changed so that only explicitly included optional dependencies are added to requirementsWithDeps
instead of including them all (I suspect that this is due to the usage of pkgMetadata.get_all or response["info"]["requires_dist"])
Yeah basically transitive deps are included by default so if your requirements.txt contains
PackageA
And PackageA depends on packageB and packageC
Then the project requirements are packageA, packageB and packageC
(Note only 1 level deep)
This is to match behaviour of poetry show. However, should probably make this optional!
@FredHappyface, I am not completely sure that this is poetry's behaviour. I mean, when you install dependencies listed in poetry you actually install all their transitive required dependencies but you will not install also optional dependencies (unless you specifically require it). On the other hand, in the lines I highlighted above, the code scans for all transitive dependencies without making any difference between actually required dependencies and optional ones. I think this is quite a wrong behavior of licensecheck since optional dependencies are not necessarily installed together with the package and thus the tool now returns some false positives like in the example at the beginning that began this thread, where an optional dependency (of pandas, i think) that was not a transient dependency of the dependencies listed in the requirements file is not compatible with the required license
@FredHappyface do you have an estimate of when a release containing this fix will be out?
Honestly depends on when I can get around to it. Should be in the next week (hopefully sooner but I'd rather not make a promise and disappoint)
@FredHappyface any updates?
Cheers for the poke! New release is avaliable on pypi / in github releases https://pypi.org/project/licensecheck/2023.4.2/
Something's still not quite right here - it seems that packages that are listed in [package.extras]
for dependencies in poetry.lock
are being included by licensecheck
, regardless of whether or not the extra was actually installed.
Example pyproject.toml
:
[tool.poetry]
name = "my-project"
version = "0.0.0"
license = "Proprietary"
description = ""
authors = []
readme = "README.md"
[tool.poetry.dependencies]
python = ">=3.9.10,<3.12"
pandas = "^2.0.1"
[tool.poetry.group.dev.dependencies]
licensecheck = "2023.4.2"
prospector = { version = "1.10.2", extras = ["with_everything"] }
[tool.licensecheck]
ignore_licenses = ["W3C License"]
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
Running licensecheck
:
Info
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃ Item ┃ Value ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ program │ licensecheck │
│ version │ 2023.1.3 │
│ license │ MIT LICENSE │
│ project_license │ PROPRIETARY LICENSE │
└─────────────────┴─────────────────────┘
List Of Packages
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Compatible ┃ Package ┃ License(s) ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ✔ │ Bottleneck │ BSD LICENSE │
│ ✔ │ Jinja2 │ BSD LICENSE │
│ ✖ │ PyQt5 │ GPL V3 │
│ ✔ │ QtPy │ MIT LICENSE │
│ ✔ │ SQLAlchemy │ MIT LICENSE │
│ ✔ │ XlsxWriter │ BSD LICENSE │
│ ✔ │ beautifulsoup4 │ MIT LICENSE │
│ ✔ │ brotlipy │ MIT │
│ ✔ │ fastparquet │ APACHE SOFTWARE LICENSE │
│ ✔ │ fsspec │ BSD LICENSE │
│ ✔ │ gcsfs │ BSD LICENSE │
│ ✔ │ html5lib │ MIT LICENSE │
│ ✖ │ hypothesis │ MOZILLA PUBLIC LICENSE 2.0 (MPL 2.0) │
│ ✔ │ lxml │ BSD LICENSE │
│ ✔ │ matplotlib │ PYTHON SOFTWARE FOUNDATION LICENSE │
│ ✔ │ numba │ BSD LICENSE │
│ ✔ │ numexpr │ MIT LICENSE │
│ ✔ │ numpy │ BSD LICENSE │
│ ✔ │ odfpy │ APACHE SOFTWARE LICENSE;; GNU GENERAL PUBLIC LICENSE (GPL);; GNU LIBRARY OR LESSER GENERAL PUBLIC LICENSE (LGPL) │
│ ✔ │ openpyxl │ MIT LICENSE │
│ ✔ │ pandas │ BSD LICENSE │
│ ✔ │ pandas-gbq │ BSD LICENSE │
│ ✔ │ psycopg2 │ GNU LIBRARY OR LESSER GENERAL PUBLIC LICENSE (LGPL) │
│ ✔ │ pyarrow │ APACHE SOFTWARE LICENSE │
│ ✔ │ pymysql │ MIT LICENSE │
│ ✔ │ pyreadstat │ APACHE SOFTWARE LICENSE │
│ ✔ │ pytest │ MIT LICENSE │
│ ✔ │ pytest-asyncio │ APACHE SOFTWARE LICENSE │
│ ✔ │ pytest-xdist │ MIT LICENSE │
│ ✔ │ python-dateutil │ BSD LICENSE;; APACHE SOFTWARE LICENSE │
│ ✔ │ python-snappy │ BSD LICENSE │
│ ✔ │ pytz │ MIT LICENSE │
│ ✔ │ pyxlsb │ GNU LESSER GENERAL PUBLIC LICENSE V3 OR LATER (LGPLV3+) │
│ ✔ │ s3fs │ BSD LICENSE │
│ ✔ │ scipy │ BSD LICENSE │
│ ✔ │ tables │ BSD LICENSE │
│ ✔ │ tabulate │ MIT LICENSE │
│ ✔ │ tzdata │ APACHE SOFTWARE LICENSE │
│ ✔ │ xarray │ APACHE SOFTWARE LICENSE │
│ ✔ │ xlrd │ BSD LICENSE │
│ ✔ │ zstandard │ BSD LICENSE │
└────────────┴─────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
It appears that PyQt5
and hypothesis
both fail. However, running poetry show
confirms that neither package is installed:
astroid 2.15.6 An abstract syntax tree for Python with inference support.
attrs 21.4.0 Classes Without Boilerplate
bandit 1.7.5 Security oriented static analyser for python code.
build 0.10.0 A simple, correct Python build frontend
cattrs 23.1.2 Composable complex class support for attrs and dataclasses.
certifi 2023.7.22 Python package for providing Mozilla's CA Bundle.
charset-normalizer 3.2.0 The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet.
dill 0.3.7 serialize all of Python
docutils 0.20.1 Docutils -- Python Documentation Utilities
dodgy 0.2.1 Dodgy: Searches for dodgy looking lines in Python code
fhconfparser 2022 Provides a config language independent way to read a config file.
flake8 2.3.0 the modular source code checker: pep8, pyflakes and co
flake8-polyfill 1.0.2 Polyfill package for Flake8 plugins
gitdb 4.0.10 Git Object Database
gitpython 3.1.32 GitPython is a Python library used to interact with Git repositories
idna 3.4 Internationalized Domain Names in Applications (IDNA)
isort 5.12.0 A Python utility / library to sort Python imports.
lazy-object-proxy 1.9.0 A fast and thorough lazy object proxy.
licensecheck 2023.4.2 Output the licenses used by dependencies and check if these are compatible with the project license
markdown-it-py 3.0.0 Python port of markdown-it. Markdown parsing, done right!
mccabe 0.7.0 McCabe checker, plugin for flake8
mdurl 0.1.2 Markdown URL utilities
mypy 1.5.1 Optional static typing for Python
mypy-extensions 1.0.0 Type system extensions for programs checked with the mypy type checker.
nodeenv 1.8.0 Node.js virtual environment builder
numpy 1.25.2 Fundamental package for array computing in Python
packaging 23.1 Core utilities for Python packages
pandas 2.0.3 Powerful data structures for data analysis, time series, and statistics
pbr 5.11.1 Python Build Reasonableness
pep8 1.7.1 Python style guide checker
pep8-naming 0.10.0 Check PEP-8 naming conventions, plugin for flake8
platformdirs 3.10.0 A small Python package for determining appropriate platform-specific dirs, e.g. a "user data dir".
prospector 1.10.2 Prospector is a tool to analyse Python code by aggregating the result of other tools.
pycodestyle 2.11.0 Python style guide checker
pydocstyle 6.3.0 Python docstring style checker
pyflakes 2.5.0 passive checker of Python programs
pygments 2.16.1 Pygments is a syntax highlighting package written in Python.
pylint 2.17.5 python code static checker
pylint-celery 0.3 pylint-celery is a Pylint plugin to aid Pylint in recognising and understandingerrors caused when using the Celery l...
pylint-django 2.5.3 A Pylint plugin to help Pylint understand the Django web framework
pylint-flask 0.6 pylint-flask is a Pylint plugin to aid Pylint in recognizing and understanding errors caused when using Flask
pylint-plugin-utils 0.7 Utilities and helpers for writing Pylint plugins
pyproject-hooks 1.0.0 Wrappers to call pyproject.toml-based build backend hooks.
pyright 1.1.324 Command line wrapper for pyright
pyroma 4.2 Test your project's packaging friendliness
python-dateutil 2.8.2 Extensions to the standard Python datetime module
pytz 2023.3 World timezone definitions, modern and historical
pyyaml 6.0.1 YAML parser and emitter for Python
requests 2.31.0 Python HTTP for Humans.
requests-cache 1.1.0 A persistent cache for python requests
requirements-detector 1.2.2 Python tool to find and list requirements of a Python project
rich 13.5.2 Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal
semver 3.0.1 Python helper for Semantic Versioning (https://semver.org)
setoptconf-tmp 0.3.1 A module for retrieving program settings from various sources in a consistant method.
setuptools 68.1.2 Easily download, build, install, upgrade, and uninstall Python packages
six 1.16.0 Python 2 and 3 compatibility utilities
smmap 5.0.0 A pure Python implementation of a sliding window memory map manager
snowballstemmer 2.2.0 This package provides 29 stemmers for 28 languages generated from Snowball algorithms.
stevedore 5.1.0 Manage dynamic plugins for Python applications
toml 0.10.2 Python Library for Tom's Obvious, Minimal Language
tomli 2.0.1 A lil' TOML parser
tomlkit 0.12.1 Style preserving TOML library
trove-classifiers 2023.8.7 Canonical source for classifiers on PyPI (pypi.org).
typing-extensions 4.7.1 Backported and Experimental Type Hints for Python 3.7+
tzdata 2023.3 Provider of IANA time zone data
url-normalize 1.4.3 URL normalization for Python
urllib3 2.0.4 HTTP library with thread-safe connection pooling, file post, and more.
vulture 2.9.1 Find dead code
wrapt 1.15.0 Module for decorators, wrappers and monkey patching.
@FredHappyface - would you like me to open a new issue for this?
Nah dw about it, I'll reopen this one and investigate - thought I'd fixed the funky behaviour here!
Can you provide info on how you are running licensecheck please? As in the actual command - I'm wondering if there's something I need to document more clearly as it sounds like you are using it to parse the poetry.lock?
Can you provide info on how you are running licensecheck please? As in the actual command - I'm wondering if there's something I need to document more clearly as it sounds like you are using it to parse the poetry.lock?
We're invoking in our CI process with python -m licensecheck --zero
, but simply running licensecheck
in a terminal results in the same behavior. We do also have configuration in pyproject.toml
under the [tool.licensecheck]
key.
@FredHappyface any ideas on this?
@FredHappyface @emesar I hit this issue too in my testing and have got a PR with details of the symptoms as well as a potential fix. Look forward to hearing your feedback. tia!
Bug
System info
-OS: Ubuntu 22.04.2 LTS -Python: 3.10.6 -licensecheck version: 2023.3
Describe the bug
While checking a requirements file, the licensecheck command's output return more packages than the ones reported in the requirements.txt.
Here I attached a zipped version of an environment to reproduce the error: example.zip
To reproduce the error:
Expected outcome
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓ ┃ Item ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩ │ program │ licensecheck │ │ version │ 2023.3.0 │ │ license │ mit │ │ project_license │ mit │ └─────────────────┴──────────────┘
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Compatible ┃ Package ┃ License(s) ┃ ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ ✔ │ annotated-types │ MIT License │ │ ✔ │ certifi │ Mozilla Public License 2.0 (MPL 2.0) │ │ ✔ │ cfg-load │ MIT License │ │ ✔ │ charset-normalizer │ MIT License │ │ ✔ │ deprecated │ MIT License │ │ ✔ │ idna │ BSD License │ │ ✔ │ mpu │ MIT License │ │ ✔ │ numpy │ BSD License │ │ ✔ │ pandas │ BSD License │ │ ✔ │ pydantic │ MIT License │ │ ✔ │ pydantic-core │ MIT License │ │ ✔ │ python-dateutil │ Apache Software License;; BSD License │ │ ✔ │ pytz │ MIT License │ │ ✔ │ pyyaml │ MIT License │ │ ✔ │ requests │ Apache Software License │ │ ✔ │ scipy │ BSD License │ │ ✔ │ six │ MIT License │ │ ✔ │ tzdata │ Apache Software License │ │ ✔ │ tzlocal │ MIT License │ │ ✔ │ urllib3 │ MIT License │ │ ✔ │ wrapt │ BSD License │ └────────────┴────────────────────┴───────────────────────────────────────┘
This are the exactly the packages reported in the requirements.txt.
Actual outcome
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Item ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ program │ licensecheck │ │ version │ 2023.1.3 │ │ license │ MIT LICENSE │ │ project_license │ NO LICENSE/ UNKNOWN LICENSE │ └─────────────────┴─────────────────────────────┘
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Compatible ┃ Package ┃ License(s) ┃ ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ ✔ │ Bottleneck │ BSD LICENSE │ │ ✔ │ Brotli │ MIT LICENSE │ │ ✔ │ Deprecated │ MIT LICENSE │ │ ✔ │ Jinja2 │ BSD LICENSE │ │ ✔ │ Pillow │ HISTORICAL PERMISSION NOTICE AND DISCLAIMER (HPND) │ │ ✖ │ PyQt5 │ GPL V3 │ │ ✔ │ PySocks │ BSD │ │ ✔ │ PyYAML │ MIT LICENSE │ │ ✔ │ QtPy │ MIT LICENSE │ │ ✔ │ SQLAlchemy │ MIT LICENSE │ │ ✔ │ Sphinx │ BSD LICENSE │ │ ✔ │ XlsxWriter │ BSD LICENSE │ │ ✔ │ annotated-types │ MIT LICENSE │ │ ✔ │ asv │ BSD LICENSE │ │ ✔ │ backports.zoneinfo │ APACHE SOFTWARE LICENSE │ │ ✔ │ beautifulsoup4 │ MIT LICENSE │ │ ✔ │ black │ MIT LICENSE │ │ ✔ │ boto3 │ APACHE SOFTWARE LICENSE │ │ ✔ │ brotlicffi │ MIT │ │ ✔ │ brotlipy │ MIT │ │ ✔ │ bump2version │ MIT LICENSE │ │ ✖ │ certifi │ MOZILLA PUBLIC LICENSE 2.0 (MPL 2.0) │ │ ✔ │ cfg-load │ MIT LICENSE │ │ ✔ │ chardet │ GNU LESSER GENERAL PUBLIC LICENSE V2 OR LATER (LGPLV2+) │ │ ✔ │ charset-normalizer │ MIT LICENSE │ │ ✔ │ check-manifest │ MIT LICENSE │ │ ✔ │ click │ BSD LICENSE │ │ ✔ │ cryptography │ APACHE SOFTWARE LICENSE;; BSD LICENSE │ │ ✔ │ cython-lint │ MIT LICENSE │ │ ✔ │ doit │ MIT LICENSE │ │ ✔ │ email-validator │ CC0 1.0 UNIVERSAL (CC0 1.0) PUBLIC DOMAIN DEDICATION │ │ ✔ │ fastparquet │ APACHE SOFTWARE LICENSE │ │ ✔ │ flake8 │ MIT LICENSE │ │ ✔ │ fsspec │ BSD LICENSE │ │ ✔ │ gcsfs │ BSD LICENSE │ │ ✔ │ gmpy2 │ GNU LESSER GENERAL PUBLIC LICENSE V3 OR LATER (LGPLV3+) │ │ ✔ │ html5lib │ MIT LICENSE │ │ ✖ │ hypothesis │ MOZILLA PUBLIC LICENSE 2.0 (MPL 2.0) │ │ ✔ │ idna │ BSD LICENSE │ │ ✔ │ jupytext │ MIT LICENSE │ │ ✔ │ lxml │ BSD LICENSE │ │ ✔ │ matplotlib │ PYTHON SOFTWARE FOUNDATION LICENSE │ │ ✔ │ mpmath │ BSD LICENSE │ │ ✔ │ mpu │ MIT LICENSE │ │ ✔ │ mypy │ MIT LICENSE │ │ ✔ │ myst-nb │ MIT LICENSE │ │ ✔ │ numba │ BSD LICENSE │ │ ✔ │ numexpr │ MIT LICENSE │ │ ✔ │ numpy │ BSD LICENSE │ │ ✔ │ numpydoc │ BSD LICENSE │ │ ✔ │ odfpy │ APACHE SOFTWARE LICENSE;; GNU GENERAL PUBLIC LICENSE (GPL);; GNU LIBRARY OR LESSER GENERAL PUBLIC LICENSE (LGPL) │ │ ✔ │ openpyxl │ MIT LICENSE │ │ ✔ │ pandas │ BSD LICENSE │ │ ✔ │ pandas-gbq │ BSD LICENSE │ │ ✔ │ pooch │ BSD LICENSE │ │ ✔ │ psycopg2 │ GNU LIBRARY OR LESSER GENERAL PUBLIC LICENSE (LGPL) │ │ ✔ │ pyOpenSSL │ APACHE SOFTWARE LICENSE │ │ ✔ │ pyarrow │ APACHE SOFTWARE LICENSE │ │ ✔ │ pycodestyle │ MIT LICENSE │ │ ✔ │ pydantic │ MIT LICENSE │ │ ✔ │ pydantic-core │ MIT LICENSE │ │ ✔ │ pydata-sphinx-theme │ BSD LICENSE │ │ ✔ │ pydevtool │ MIT LICENSE │ │ ✔ │ pymysql │ MIT LICENSE │ │ ✔ │ pyreadstat │ APACHE SOFTWARE LICENSE │ │ ✔ │ pyroma │ MIT LICENSE │ │ ✔ │ pytest │ MIT LICENSE │ │ ✔ │ pytest-asyncio │ APACHE SOFTWARE LICENSE │ │ ✔ │ pytest-cov │ MIT LICENSE │ │ ✔ │ pytest-flake8 │ BSD LICENSE │ │ ✔ │ pytest-mccabe │ MIT LICENSE │ │ ✔ │ pytest-mock │ MIT LICENSE │ │ ✔ │ pytest-timeout │ DFSG APPROVED;; MIT LICENSE │ │ ✔ │ pytest-xdist │ MIT LICENSE │ │ ✔ │ python-dateutil │ APACHE SOFTWARE LICENSE;; BSD LICENSE │ │ ✔ │ python-magic │ MIT LICENSE │ │ ✔ │ python-snappy │ BSD LICENSE │ │ ✔ │ pytz │ MIT LICENSE │ │ ✔ │ pyxlsb │ GNU LESSER GENERAL PUBLIC LICENSE V3 OR LATER (LGPLV3+) │ │ ✔ │ requests │ APACHE SOFTWARE LICENSE │ │ ✔ │ rich-click │ MIT LICENSE │ │ ✔ │ ruff │ MIT LICENSE │ │ ✔ │ s3fs │ BSD LICENSE │ │ ✔ │ scikit-umfpack │ BSD LICENSE │ │ ✔ │ scipy │ BSD LICENSE │ │ ✔ │ simplejson │ ACADEMIC FREE LICENSE (AFL);; MIT LICENSE │ │ ✔ │ six │ MIT LICENSE │ │ ✔ │ sphinx_design │ MIT LICENSE │ │ ✔ │ tables │ BSD LICENSE │ │ ✔ │ tabulate │ MIT LICENSE │ │ ✔ │ threadpoolctl │ BSD LICENSE │ │ ✔ │ tox │ MIT LICENSE │ │ ✔ │ types-psutil │ APACHE SOFTWARE LICENSE │ │ ✔ │ typing_extensions │ PYTHON SOFTWARE FOUNDATION LICENSE │ │ ✔ │ tzdata │ APACHE SOFTWARE LICENSE │ │ ✔ │ tzlocal │ MIT LICENSE │ │ ✔ │ urllib3 │ MIT LICENSE │ │ ✔ │ urllib3-secure-extra │ MIT LICENSE │ │ ✔ │ wrapt │ BSD LICENSE │ │ ✔ │ xarray │ APACHE SOFTWARE LICENSE │ │ ✔ │ xlrd │ BSD LICENSE │ │ ✖ │ zest.releaser │ GNU GENERAL PUBLIC LICENSE (GPL) │ │ ✔ │ zstandard │ BSD LICENSE │ └────────────┴──────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
The main issue is that there are more packages than the ones reported in the requirements file but also the version of licensecheck in the header is wrong.