Open m000 opened 2 years ago
di
commented
2 years ago This is related to https://bugs.debian.org/994952: Debian has a patch to virtualenv to explicitly install pkg_resources (as it's a separate binary package and wheel, from setuptools, in Debian), but it doesn't exist on PyPI. This doesn't affect you when using python -m venv because this isn't using the patched virtualenv.
I suspect specifying the -l/--local flag to pip-audit might fix this for you.
woodruffw
commented
2 years ago Dates back to at least 2018 as well: https://stackoverflow.com/questions/38992194/why-does-pip-freeze-list-pkg-resources-0-0-0
I'm not sure there's much pip-audit can do here: we're faithfully retrieving the metadata that the environment gives us, which for some Linux distros is just incorrect and can't be reconciled.
m000
commented
2 years ago @di Yes, -l sounds promising, but it made no difference.
@woodruffw Seems relevant, but not exactly. I don't get pkg-resources in my pip listing, only setuptools:
$ pip freeze --all
pip==20.3.4
setuptools==52.0.0
wheel==0.34.2I was wondering, are there any other packages that use 0.0.0 as version? This seems like a legit heuristic for excluding a package from the dependencies list.
Other than that, adding some sort of --ignore option could be a workaround. This could also come handy in other situations. E.g. run pip-audit as part of your CI/CD pipeline, but don't fail for packages that you know that have vulnerabilities, but you can't upgrade for one reason or another.
I can prob implement either of these and send a PR if needed.
woodruffw
commented
2 years ago PEP 440 says the following:
All numeric components MAY be zero. Except as described below for the release segment, a numeric component of zero has no special significance aside from always being the lowest possible value in the version ordering.
...which unfortunately means that publishing a 0.0.0 distribution on PyPI is perfectly valid, and dependency resolvers must respect and handle it correctly. So that heuristic won't be safe, unfortunately.
Other than that, adding some sort of
--ignoreoption could be a workaround. This could also come handy in other situations. E.g. run pip-audit as part of your CI/CD pipeline, but don't fail for packages that you know that have vulnerabilities, but you can't upgrade for one reason or another.
Yep! #209 also raises this idea; I think it's something we want, but we'll want to flesh out a reasonable design first. In particular, allowing users to ignore arbitrary packages has the potential to make pip-audit -r perform even more unreliably than it currently does, so we'll want to be careful about how we allow ignoring.
m000
commented
2 years ago pkg-resources is a weird case. It has an empty link list page (HTTP 200) but no project page (HTTP 404).
We want to switch from Safety to pip-audit, so this was really a show-stopper. I have an incoming PR that adds a flag to the cli for skipping unresolvable packages. Debian is an important distribution, so it would probably be for the benefit of the project to have a workaround for this quirk. But if you prefer to wait for a better solution, it can happily live in my fork.
The PR could perhaps be a bit more specialized by having get_project_from_pypi() raise a PyPINotFoundError or a (new) PyPINoProjectLinksError if it can find no links in the links page.
woodruffw
commented
2 years ago pkg-resources is a weird case. It has an empty link list page (HTTP 200) but no project page (HTTP 404).
Yeah, it's strange that it has a simple index page at all -- pkg-resources is a wholly-contained sub-package of setuptools, so it really shouldn't be independently listed/indexed at all.
Thanks for making the PR! I'll take a look now.
m000
commented
2 years ago Let me know what you think. Also looking to #197, it is probably better to create a new exception so we can tell different case of failed resolution apart.
woodruffw
commented
2 years ago Just for future visiblity: here are the xrefs for the same pkg_resources=0.0.0 pain in pip: https://github.com/pypa/pip/issues/8331 https://github.com/pypa/pip/issues/4022
woodruffw
commented
2 years ago Just looping back on this: if you have a fully resolved requirements.txt, passing --no-deps might avoid this error for you (since it won't attempt to do any extra resolution).
richardash1981
commented
2 years ago Running pip-audit on Ubuntu seems to break lots of things which work fine in other environments (I am developing on Gentoo but with an Ubuntu CI environment). What works in Ubuntu 18.04 and 20.04 is as follows:
python3 -m venv .venv
source .venv/bin/activate
pip install pip-audit
pip-audit --strict -r requirements.txtInserting the extra layer between the OS and pip-audit seems to be necessary to get rid of interference from the Ubuntu packaging (not just this issue, but also a bunch of packages with no upstream on pypi etc.).
woodruffw
commented
2 years ago Yeah, virtual environments are strongly recommended in general, but particularly on Ubuntu/Debian, where the upstream maintainers continue to debundle core parts of Python.
Assuming that you're following the other best practices for Python packaging and deployment, your project should already have its own virtual environment. If that's the case, then you can do this instead:
(env) $ python -m pip install pip-audit
(env) $ pip-audit...and that'll automatically audit the current (virtual) environment, which will also be much faster than having to re-do resolution in requirements mode (-r requirements.txt).
di
commented
2 years ago Note that https://peps.python.org/pep-0668/ might eventually help with this, but it'll probably be a while.
Bug description
When running pip-audit with some requirements files, it fails to complete because bogus requirement
pkg_resources==0.0.0is introduced by pypi_provider. E.g.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=<Requirement('pkg_resources==0.0.0')>, parent=<django-admin-inline-paginator==0.2.2 wheel=False>)]If the requirements are installed in a virtualenv and pip-audit is run from the virtualenv without further arguments, it works as expected.
Reproduction steps
Expected behavior
It should work in both cases.
Screenshots and logs
Command output:
Verbose output:
Platform information
pip-auditversion (pip-audit -V): pip-audit 1.1.1python -Vorpython3 -V): Python 3.9.2pipversion (pip -Vorpip3 -V): pip 21.3.1 from /root/.local/lib/python3.9/site-packages/pip (python 3.9)Additional context
The exception trace is in resolvelib. But I've tracked down where the bogus dependency is generated. It boils down to this function in resolvers.py. Which from what I understand is implemented by pypi_provider.py of pip-audit.