search

jazzband / pip-tools

A set of tools to keep your pinned Python dependencies fresh.
https://pip-tools.rtfd.io
BSD 3-Clause "New" or "Revised" License
7.6k stars 607 forks source link

Error when supplying an existing file `ModuleNotFoundError: No module named 'setuptools.extern.six'` #2109

Closed darrencl closed 2 weeks ago

darrencl commented 2 weeks ago

Environment Versions

  1. OS Type: Ubuntu 20.04
  2. Python version: $ python -V Python 3.12.4
  3. pip version: $ pip --version 24.1.1
  4. pip-tools version: $ pip-compile --version 7.4.1

Steps to replicate

My package has both requirements.txt and requirements_dev.txt. Here is what I did:

  1. Update requirement versions in setup.py
  2. Run pip-compile setup.py --extra dev -o requirements_dev.txt

When I did that, below error shows up, which I think is related to click package.

This, however, will not happen if I supply a new output file (e.g., test_req_dev.txt). For example, below command will run just fine.

pip-compile setup.py --extra dev -o test_req_dev.txt

I am unsure if this is an intended behavior. If so, the error should be handled to produce clearer error message to user.

Expected result

Overwrite requirements_dev.txt with an updated version.

Actual result

Error with click

(suspect312) darren@darren-Latitude-7490:~/project/mrspectroproc$ pip-compile setup.py -o requirements.txt 
Warning: the ECDSA host key for 'github.com' differs from the key for the IP address '20.205.243.166'
Offending key for IP in /home/darren/.ssh/known_hosts:18
Matching host key in /home/darren/.ssh/known_hosts:21
Are you sure you want to continue connecting (yes/no)? yes
Warning: the ECDSA host key for 'github.com' differs from the key for the IP address '20.205.243.166'
Offending key for IP in /home/darren/.ssh/known_hosts:18
Matching host key in /home/darren/.ssh/known_hosts:21
Are you sure you want to continue connecting (yes/no)? yes 
Traceback (most recent call last):
  File "/home/darren/anaconda3/envs/suspect312/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/piptools/scripts/compile.py", line 470, in cli
    results = resolver.resolve(max_rounds=max_rounds)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/piptools/resolver.py", line 604, in resolve
    is_resolved = self._do_resolve(
                  ^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/piptools/resolver.py", line 636, in _do_resolve
    resolver.resolve(
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
    return bool(self._sequence)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 174, in __bool__
    return any(self)
           ^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 162, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 53, in _iter_built
    candidate = func()
                ^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 185, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 231, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 303, in __init__
    super().__init__(
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 235, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 314, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 527, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 642, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 72, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 56, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 126, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 103, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_internal/utils/misc.py", line 709, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darren/anaconda3/envs/suspect312/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/tmp/pip-build-env-gi9g6d6m/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 18, in <module>
    from setuptools.extern.six import PY3, string_types
ModuleNotFoundError: No module named 'setuptools.extern.six'
chrysle commented 2 weeks ago

Looks like an outdated setuptools is being used (Python 2 compatibility via six was removed in v47.0.0). Could you share the contents of your in- and outfiles?

darrencl commented 2 weeks ago

Hi @chrysle ,

Thanks for getting back to this quickly!

In relation to my setuptools it seems to use the updated version unless somehow. 

(suspect312) darren@darren-Latitude-7490:~/project/mrspectroproc$ python -m pip show setuptools
Name: setuptools
Version: 70.1.1
Summary: Easily download, build, install, upgrade, and uninstall Python packages
Home-page: https://github.com/pypa/setuptools
Author: 
Author-email: Python Packaging Authority <distutils-sig@python.org>
License: 
Location: /home/darren/anaconda3/envs/suspect312/lib/python3.12/site-packages
Requires: 
Required-by: pip-tools

I had a fiddle with this after submitting this issue and this now strangely works for me. If I remember correctly, what I did was uninstalling the pip-tools in the conda base environment. I am not sure how is that related though, since the error was clearly from the "/home/darren/anaconda3/envs/suspect312/bin/pip-compile", not the base environment path. It might be something else I did, I couldn't remember clearly.

However, I can confirm I did update the setuptools (i.e., python -m pip install -U setuptools) as well and still had issues. As you can see, I submitted this issue at 11.31AM (GMT+7) and my setuptools was updated at 10.31AM (coincidentally, the difference is exactly 1 hour! :smile:) .

In [11]: for dist in distributions():
    ...:     if dist.metadata["Name"] == "setuptools":
    ...:         print("%s %s: %s" % (dist.metadata["Name"], dist.version, time.ctime(os.path.getctime(dist._path))))
    ...: 
setuptools 70.1.1: Thu Jun 27 10:31:33 2024

In [12]: time.ctime()
Out[12]: 'Thu Jun 27 14:55:38 2024'

My setup.py is a standard one with install_requires and extras_require.

Anyway, I have always been using pip-compile for this package, but it was a couple of years back since I updated all dependencies, including pip-tools, and then I got this weird error despite updating setuptools.

I think I will close this for now since I don't have this issue anymore.