[24.1] `pip install py_find_1st` fails with 24.1, whereas 24.0 does not

[xmatthias]

Description

When installing a package - py_find_1st - the install is failing with pip 24.1 - but works with 24.0.

The error happens with

  File "/usr/lib/python3.12/tarfile.py", line 2725, in _find_link_target
    raise KeyError("linkname %r not found" % linkname)
KeyError: "linkname 'py_find_1st-1.1.6/README' not found"

it's unclear to me why this would work in 24.0 - but fail in 24.1. The package uses a link from readme.md to readme - but i don't think that's disallowed?

~ tar tzvpf py_find_1st-1.1.6.tar.gz 
drwxr-xr-x roebel/staff      0 2023-10-28 18:01 py_find_1st-1.1.6/
-rw-r--r-- roebel/staff   7156 2023-10-28 17:15 py_find_1st-1.1.6/LONG_DESCR
-rw-r--r-- roebel/staff   8020 2023-10-28 18:01 py_find_1st-1.1.6/PKG-INFO
-rw-r--r-- roebel/staff   6896 2023-10-28 17:15 py_find_1st-1.1.6/README
hrw-r--r-- roebel/staff      0 2023-10-28 17:15 py_find_1st-1.1.6/README.md link to py_find_1st-1.1.6/README
-rw-r--r-- roebel/staff    159 2021-02-02 13:17 py_find_1st-1.1.6/pyproject.toml
-rw-r--r-- roebel/staff     34 2023-10-28 17:54 py_find_1st-1.1.6/requirements.txt
-rw-r--r-- roebel/staff     40 2019-06-28 23:32 py_find_1st-1.1.6/setup.cfg
-rw-r--r-- roebel/staff   6570 2023-10-28 17:54 py_find_1st-1.1.6/setup.py
drwxr-xr-x roebel/staff      0 2023-10-28 18:01 py_find_1st-1.1.6/test/
-rwxr-xr-x roebel/staff   1856 2023-10-28 17:10 py_find_1st-1.1.6/test/test_find_1st.py
drwxr-xr-x roebel/staff      0 2023-10-28 18:01 py_find_1st-1.1.6/utils_find_1st/
-rw-r--r-- roebel/staff    305 2023-10-28 18:01 py_find_1st-1.1.6/utils_find_1st/__init__.py
-rw-r--r-- roebel/staff   5475 2019-08-04 17:17 py_find_1st-1.1.6/utils_find_1st/find_1st.cpp

The changelog doesn't indicate something that would break like this ... unless i've missed that point ?

Expected behavior

Works on 24.0 and 24.1

pip version

24.1

Python version

3.12

OS

ubuntu

How to Reproduce

apt install build-essential
pip install pip==24.1
pip install py_find_1st
# Notice it failing
pip install pip==24.0
pip install py_find_1st

Output

...
  File "/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 185, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 231, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 303, in __init__
    super().__init__(
  File "/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 235, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 314, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 527, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 598, in _prepare_linked_requirement
    local_file = unpack_url(
                 ^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 180, in unpack_url
    unpack_file(file.path, location, file.content_type)
  File "/.venv/lib/python3.12/site-packages/pip/_internal/utils/unpacking.py", line 316, in unpack_file
    untar_file(filename, location)
  File "/.venv/lib/python3.12/site-packages/pip/_internal/utils/unpacking.py", line 235, in untar_file
    tar.extractall(location, filter=pip_filter)
  File "/usr/lib/python3.12/tarfile.py", line 2269, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.12/tarfile.py", line 2332, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.12/tarfile.py", line 2423, in _extract_member
    self.makelink(tarinfo, targetpath)
  File "/usr/lib/python3.12/tarfile.py", line 2521, in makelink
    self._extract_member(self._find_link_target(tarinfo),
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/tarfile.py", line 2725, in _find_link_target
    raise KeyError("linkname %r not found" % linkname)
KeyError: "linkname 'py_find_1st-1.1.6/README' not found"

Code of Conduct

• [X] I agree to follow the PSF Code of Conduct.

[ichard26]

We did update our tar unpacking logic to rely more on the stdlib implementation to benefit from tarfile data filters (https://github.com/pypa/pip/pull/12214). I don't have access to a development environment so I won't be able to debug this until later today, but for now, you can downgrade pip to 24.0.

[notatallshaw]

FYI I can reproduce with python -m pip install --dry-run py_find_1st

And the exception in question hasn't recently changed: https://github.com/python/cpython/blame/v3.12.4/Lib/tarfile.py#L2725

And Python can extract the tar file:

$ wget https://files.pythonhosted.org/packages/1d/24/4b4dc8fcf9ad5e6ac8ac6c6bd4be8100f07846c7a7ce6aa9c44ee1eaa429/py_find_1st-1.1.6.tar.gz
$ python -m tarfile -e py_find_1st-1.1.6.tar.gz

So it's probably related to the filtering feature.

[henryiii]

FWIW, python -m tarfile --filter data -e py_find_1st-1.1.6.tar.gz seems fine too. Filtering added in https://github.com/pypa/pip/pull/12214. Ahh, but it's wrapped with "pip specific" behavior. And I don't see a test to make sure good symlinks work, just tests for rejecting bad symlinks.

[notatallshaw]

I did a little investigation but didn't have time to post my results. The issue seems to be related to this tar file has a hard link that points from README.md to README.

I wasn't able to find another example of a real Python package that had the same problem.

It was very likely caused by https://github.com/pypa/pip/pull/12214 if @encukou you want to take a look.

[shahidhafiz]

I had the same issue until I ran it using powershell as admin. worked first time.

[gneil90]

getting the same on osx

[apenney]

We use pandocfilters==1.4.1 and that exhibits the problem if you need a real world example of this breaking in the wild!

[pfmoore]

The pandocfilters 1.4.1 sdist seems to be broken:

Note that the target of README.rst is pandocfilters-1.4.1/README, which does not exist as the error message says.

The same is true of py_find_1st, which suggests that this might be a build backend bug creating an invalid sdist.

I don't think there's a pip issue here (even though things are reported as working in 24.0, I'd argue that's a bug in 24.0 rather than a regression).

Edit: Although on further reflection, this may depend on how a relative target for a hard link in a tar file is meant to be interpreted - as relative to the root of the tarfile, or as relative to the link. I don't know the answer to that, or even if there is a standard answer.

[pfmoore]

Ah. @henryiii pointed out here that we have a filter with pip-specific behaviour. I suspect the issue is that our filter is checking for symlinks pointing out of the tarfile, but it's using the same logic for hard links, and not taking into account the fact that symlinks are relative to the link itself, whereas hardlinks are relative to the tarfile root (as I noted above).

That sounds like a bit of a mess to sort out. @encukou as the author of that change, can you take a look?

[encukou]

I'll look into it.

[mdhiggins]

Seeing this on an older package qtfaststart as well, though it seems to be impacted by the ---no-cache-dir parameter though this previously worked fine on 24.0

https://github.com/mdhiggins/sma-mod/issues/17 https://github.com/mdhiggins/sickbeard_mp4_automator/issues/1716

[EgyptianM]

I have the same issue while installing Freqtrade on Linux Mint

  Downloading py_find_1st-1.1.6.tar.gz (8.6 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 179, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
    return bool(self._sequence)
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 174, in __bool__
    return any(self)
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 162, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 53, in _iter_built
    candidate = func()
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 185, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 231, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 303, in __init__
    super().__init__(
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
    self.dist = self._prepare()
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 235, in _prepare
    dist = self._prepare_distribution()
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 314, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 527, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 598, in _prepare_linked_requirement
    local_file = unpack_url(
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 180, in unpack_url
    unpack_file(file.path, location, file.content_type)
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/utils/unpacking.py", line 316, in unpack_file
    untar_file(filename, location)
  File "/home/mohammad/Downloads/freqtrade/.venv/lib/python3.10/site-packages/pip/_internal/utils/unpacking.py", line 235, in untar_file
    tar.extractall(location, filter=pip_filter)
  File "/usr/lib/python3.10/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.10/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.10/tarfile.py", line 2411, in _extract_member
    self.makelink(tarinfo, targetpath)
  File "/usr/lib/python3.10/tarfile.py", line 2508, in makelink
    self._extract_member(self._find_link_target(tarinfo),
  File "/usr/lib/python3.10/tarfile.py", line 2712, in _find_link_target
    raise KeyError("linkname %r not found" % linkname)
KeyError: "linkname 'py_find_1st-1.1.6/README' not found"

[sanikeit]

I've also encountered the same issue when trying to install the py-find-1st package. The error message I receive is:

KeyError: "linkname 'py_find_1st-1.1.6/README' not found"

The temporary workaround by downgrading pip to version 24.0, also resolved the issue for me.

[encukou]

This should be fixed by https://github.com/pypa/pip/pull/12799.