[BUG] 61.0.0 breaks "setup.py install" by missing dependencies in install_requires

[andy-maier]

setuptools version

61.0.0

Python version

3.10

OS

ubuntu-latest in GitHub Actions

Additional environment information

No response

Description

In our pywbem package, we test the different documented methods to install the package. One of them is still "setup.py install". I do understand that it is deprecated since setuptools v58.3.0, but that still means it should be expected to work.

The issue is that the "setup.py install" command successfully installs the pywbem package, but upon import it turns out that dependent packages were not installed, e.g. "six".

The GitHub Actions test run showing that is: https://github.com/pywbem/pywbem/runs/5687824533?check_suite_focus=true

"pip install" on the package works with setuptools 61.0.0 and does install the dependent packages.

The requirements.txt file of the pywbem project does specify the dependent packages and the setup.py script loads the dependencies from requirements.txt and specifies them in the "install_requires" parameter of setup().

The "setup.py install" approach worked before setuptools 61.0.0, e.g. with 60.10.0, see this test run.

Expected behavior

61.0.0 should still support "setup.py install" as before.

How to Reproduce

1. Create a new virtual env on Python 3.10
2. pip install setuptools==61.0.0; pip uninstall six
3. git clone https://github.com/pywbem/pywbem.git; cd pywbem
4. ./setup.py install # should succeed
5. pip list # should display:

   Package    Version
   ---------- ----------
   pip        22.0.4
   pywbem     1.5.0.dev1
   setuptools 61.0.0
   wheel      0.37.1

   and the issue is that the dependent packages are not installed.

6. pip install . # should succeed
7. pip list # should display the correctly installed dependencies:

   Package            Version
   ------------------ ----------
   certifi            2021.10.8
   charset-normalizer 2.0.12
   idna               3.3
   nocasedict         1.0.2
   nocaselist         1.0.4
   pip                22.0.4
   ply                3.11
   pywbem             1.5.0.dev1
   PyYAML             6.0
   requests           2.27.1
   setuptools         61.0.0
   six                1.16.0
   urllib3            1.26.9
   wheel              0.37.1
   yamlloader         1.1.0

Output

See above

[andy-maier]

Probably related to #3196

[servoz]

I had the same problem this morning since I upgraded to setuptools 61.0.0 (no dependencies installed). The problem is clearly coming from the last release because if we do : pip install --upgrade setuptools==60.10.0 everything works fine!

[abravalheri]

Hi @andy-maier, thank you very much for reporting that. I checked pywbem against the fix proposed for 3196, and it seems that the issue might be different. I am still investigating the problem.

[matteius]

Similar problem -- 61.0.0 broke a test in the pipenv CI that checks that when installing this setup.py it gets the sibling package which it no longer does. https://github.com/techalchemy/test-project/blob/master/parent_folder/pep508-package/setup.py#L15

[matteius]

@abravalheri The pipenv CI just pulled in 61.1.1 but it still seems to be failing on that setupy.py line from my prior comment, ref: https://github.com/pypa/pipenv/runs/5703684724?check_suite_focus=true

Note: the python 3.6 build is passing because it cannot install such a new setuptools, and despite my best efforts to pin the setup.py, CI, and run-tests.sh it was still pulling in the latest setuptools, perhaps pip was doing this.

Please let me know if I should open a new ticket for this issue or if you have any other suggestions.

[abravalheri]

Thanks @matteius, I will have a look.

[abravalheri]

Hi @matteius, please note that the problem seems to have been happening since before 61.0.0:

% git clone https://github.com/techalchemy/test-project
% cd test-project/parent_folder/pep508-package
% virtualenv .venv
% .venv/bin/python -m pip install -U pip setuptools==60.10.0 wheel
% .venv/bin/python -m pip list
Package    Version
---------- -------
pip        22.0.4
setuptools 60.10.0
wheel      0.37.1
% .venv/bin/python setup.py install
...
Searching for sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...
Couldn't find index page for 'sibling_package' (maybe misspelled?)
...
No local packages or working download links found for sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
error: Could not find suitable distribution for Requirement.parse('sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package')

Do you known when was the last time this exact test worked before? Which was the version of setuptools you were using?

BTW, I think the issue you are facing is not the same as the one reported originally here... I am even under the impression that setuptools never did manage requirements URL in the form of req @ git+https://... (but I might be wrong here).

[matteius]

@abravalheri the test was passing three days ago. I think the reason pinning setup tools doesn't fix it is because internal pip is installing the latest setup tools anyway I think. If you look at the Python 3.6 build it is still passing because it will never install the new version of setup tools since it doesn't support py3.6. I am skiing today but can look more into it tomorrow. I do believe it is caused by setup tools upgrade at this point.

[abravalheri]

Hi @matteius please do enjoy your leisure time, we can have a look at this later :)

[abravalheri]

I will just add some notes here so when you have the time you can have a look.

---

I do believe it is caused by setup tools upgrade at this point.

That is completely possible. It might very well be that v61.0.0...v61.1.1 introduced something that is causing the problem. However this seems to be a second issue, and not the one originally reported here.

I think the reason pinning setup tools doesn't fix it is because internal pip is installing the latest setup tools anyway.

I don't understand what you mean here. The test I did in my machine does not use pip at all. The original problem mentioned in this issue is related to invoking python setup.py install, so I tried to replicate it locally with the repository you linked. python setup.py install does not use pip (it uses easy_install internally).

If I instead run:

% git clone https://github.com/techalchemy/test-project
% cd test-project/parent_folder/pep508-package
% virtualenv .venv
% .venv/bin/python -m pip install -U pip setuptools==61.1.1 wheel
% .venv/bin/python -m pip list
Package    Version
---------- -------
pip        22.0.4
setuptools 61.1.1
wheel      0.37.1
% .venv/bin/python -m pip install -e .
...
  Running command git clone --filter=blob:none --quiet https://github.com/techalchemy/test-project.git /tmp/pip-install-fapb6ksm/sibling-package_2bc442202d024507bd538a882a35d28a
...
Successfully installed pep508-package-1.0.0 sibling_package-1.0.0 six-1.16.0 toml-0.10.2 urllib3-1.26.9
# I also tried `pip install .` and the outcome is the same

Then both packages seem to install fine.

If instead I run (replacing install with develop:

% rm -rf .venv
% .venv/bin/python -m pip install -U pip setuptools==60.10.0 wheel
% .venv/bin/python -m pip list
Package    Version
---------- -------
pip        22.0.4
setuptools 60.10.0
wheel      0.37.1
% .venv/bin/python setup.py develop
...
Couldn't find index page for 'sibling_package' (maybe misspelled?)
...
No local packages or working download links found for sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
error: Could not find suitable distribution for Requirement.parse('sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package')

Then I see the same problem with v60.10.0

If you manage to find a minimal reproducer along the lines of the examples before that works for setuptools==60.10.0 but does not work for setuptools==61.1.1 that would help a lot to understand this problem.

---

BTW, I am testing on:

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal

Python 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0]

[matteius]

@abravalheri Thanks for the helpful feedback. I am back now and trying to figure this out again. So pip uses setuptools, and following this pip documentation: https://pip.pypa.io/en/stable/topics/vcs-support/ lead me to a working pip install command: pip install -e "git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package"

matteius@matteius-VirtualBox:~/pipenv-triage/setuptools-3198/test-project/parent_folder/pep508-package$ pip install -e "git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package"
Defaulting to user installation because normal site-packages is not writeable
Obtaining pep508_package from git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package
  Updating ./src/pep508-package clone (to revision master)
  Running command git fetch -q --tags
  Running command git reset --hard -q 4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648
  Preparing metadata (setup.py) ... done
Collecting sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
  Cloning https://github.com/techalchemy/test-project.git (to revision master) to /tmp/pip-install-abye4d5v/sibling-package_0bce244a0b6649faa4d3ff6d1f7e36d0
  Running command git clone --filter=blob:none --quiet https://github.com/techalchemy/test-project.git /tmp/pip-install-abye4d5v/sibling-package_0bce244a0b6649faa4d3ff6d1f7e36d0
  Resolved https://github.com/techalchemy/test-project.git to commit 4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648
  Preparing metadata (setup.py) ... done
Requirement already satisfied: six in /home/matteius/.local/lib/python3.9/site-packages (from pep508_package) (1.16.0)
Requirement already satisfied: toml in /home/matteius/.local/lib/python3.9/site-packages (from sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package->pep508_package) (0.10.2)
Requirement already satisfied: urllib3 in /home/matteius/.local/lib/python3.9/site-packages (from sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package->pep508_package) (1.26.8)
Building wheels for collected packages: sibling_package
  Building wheel for sibling_package (setup.py) ... done
  Created wheel for sibling_package: filename=sibling_package-1.0.0-py3-none-any.whl size=1518 sha256=73436892f4004a44600263189c2da681355968693c5417dd29f7bcd84b1abeb6
  Stored in directory: /tmp/pip-ephem-wheel-cache-3_di4jil/wheels/de/21/b8/e6c31ccdbf6afe5ab53d4ae0a0ae80a4f93c29e14828123eb6
Successfully built sibling_package
Installing collected packages: sibling_package, pep508_package
  Attempting uninstall: pep508_package
    Found existing installation: pep508-package 1.0.0
    Uninstalling pep508-package-1.0.0:
      Successfully uninstalled pep508-package-1.0.0
  Running setup.py develop for pep508_package
Successfully installed pep508_package-1.0.0 sibling_package-1.0.0

Now I try again with setuptools-61.0.0 ... it also seems to work.

matteius@matteius-VirtualBox:~/pipenv-triage/setuptools-3198/test-project/parent_folder/pep508-package$ pip install setuptools==61.0.0
Defaulting to user installation because normal site-packages is not writeable
Collecting setuptools==61.0.0
  Downloading setuptools-61.0.0-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 5.6 MB/s eta 0:00:00
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 59.8.0
    Uninstalling setuptools-59.8.0:
      Successfully uninstalled setuptools-59.8.0
Successfully installed setuptools-61.0.0

:atest setuptools does seem to work with pip in isolation, and here is an example with no setuptools first that show it is in fact setuptools it is using to do the install. So now I am very perplexed how our test started failing with no code changes.

matteius@matteius-VirtualBox:~/pipenv-triage/setuptools-3198/test-project/parent_folder/pep508-package$ pip install -e "git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package"
Defaulting to user installation because normal site-packages is not writeable
Obtaining pep508_package from git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package
  Updating ./src/pep508-package clone (to revision master)
  Running command git fetch -q --tags
  Running command git reset --hard -q 4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
matteius@matteius-VirtualBox:~/pipenv-triage/setuptools-3198/test-project/parent_folder/pep508-package$ pip install setuptools --user
Collecting setuptools
  Using cached setuptools-61.1.1-py3-none-any.whl (1.1 MB)
Installing collected packages: setuptools
Successfully installed setuptools-61.1.1
matteius@matteius-VirtualBox:~/pipenv-triage/setuptools-3198/test-project/parent_folder/pep508-package$ pip install -e "git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package"
Defaulting to user installation because normal site-packages is not writeable
Obtaining pep508_package from git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package
  Updating ./src/pep508-package clone (to revision master)
  Running command git fetch -q --tags
  Running command git reset --hard -q 4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648
  Preparing metadata (setup.py) ... done
Collecting sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
  Cloning https://github.com/techalchemy/test-project.git (to revision master) to /tmp/pip-install-gkrre310/sibling-package_3cbedeaea7da4caabd0f8df1b4384bbf
  Running command git clone --filter=blob:none --quiet https://github.com/techalchemy/test-project.git /tmp/pip-install-gkrre310/sibling-package_3cbedeaea7da4caabd0f8df1b4384bbf
  Resolved https://github.com/techalchemy/test-project.git to commit 4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648
  Preparing metadata (setup.py) ... done
Requirement already satisfied: six in /home/matteius/.local/lib/python3.9/site-packages (from pep508_package) (1.16.0)
Requirement already satisfied: toml in /home/matteius/.local/lib/python3.9/site-packages (from sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package->pep508_package) (0.10.2)
Requirement already satisfied: urllib3 in /home/matteius/.local/lib/python3.9/site-packages (from sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package->pep508_package) (1.26.8)
Installing collected packages: pep508_package
  Attempting uninstall: pep508_package
    Found existing installation: pep508-package 1.0.0
    Uninstalling pep508-package-1.0.0:
      Successfully uninstalled pep508-package-1.0.0
  Running setup.py develop for pep508_package
Successfully installed pep508_package-1.0.0

Thanks for your help, I am going to research if any other related packages would have bumped around the same time.

[abravalheri]

Hi @matteius

here is an example with no setuptools first that show it is in fact setuptools it is using to do the install

Please note that when you run pip install -e ., a few separated steps will be executed (it is not like pip delegates everything to setuptools):

1. Pip uses setuptools to list the packages dependencies (I might be wrong, but I think it does this via setuptools.build_meta.get_requires_for_build_wheel)
2. Pip installs the dependencies
3. Pip runs python setup.py develop

Setuptools itself is not able to handle VCS URLs as dependencies. If python setup.py develop or python setup.py install is used directly, without pip adding the dependencies first, then the editable installation is expected to fail, as shown in my previous comments.

Any chance pip install -e . got replaced somewhere to python setup.py develop/install?

[matteius]

Any chance pip install -e . got replaced somewhere to python setup.py develop/install?

Doesn't seem to be -- I put a bunch of time into this last night and the perplexing thing is the breakage happened without any changes being pushed to pipenv repository. I have summarized some additional notes here, as there seems to be a similar issue encountered in pip-tools (which we do not rely upon, but whatever upstream thing it is seemed to come about at the same time for both projects): https://github.com/jazzband/pip-tools/issues/1603#issuecomment-1079831628

The telling thing to me is that our python 3.6 continues to pass for this test case, and I can see in the CI that is using an older version of setuptools because the newer versions aren't python 3.6 compatible. Despite my best efforts to pin setuptools for the other python versions, it seems to all be for not -- I think pip is bringing in the new version anyway when it goes to resolve. https://github.com/pypa/pipenv/runs/5708004672?check_suite_focus=true#step:7:38

Pip uses setuptools to list the packages dependencies (I might be wrong, but I think it does this via setuptools.build_meta.get_requires_for_build_wheel)

Yes well at least pipenv does this, it is my latest understanding too it uses that method in some places as well as prepare_metadata_for_build_wheel.

I am continuing to investigate.

[matteius]

@abravalheri I have been comparing internals of the last pipenv version that still works for this case on newer python versions 2021.5.29 with the newer versions. There is a code path that gets setup info, and what I see is in the working copy, we are getting in the metadata a valid version and requires attribute, but in the newer versions they are both empty in the metadata. I wonder if it the metadata has empty version and requires because the name is all weird in the new version 'parent-folder.sibling-package.src.sibling-package' when it should just be pep508-package so maybe it is a key mapping issue somehow now.

SETUP_INFO NOW: SetupInfo(name='pep508_package', base_dir='/tmp/pipenv-fq_39y0k-src/pep508_package', _version='0.0.0', 
_requirements=frozenset(), build_requires=(), build_backend='setuptools.build_meta:__legacy__', setup_requires=(), 
python_requires=None, _extras_requirements=(), setup_cfg=PosixPath('/tmp/pipenv-fq_39y0k-src/pep508_package/parent_folder/pep508-package/setup.cfg'), 
setup_py=PosixPath('/tmp/pipenv-fq_39y0k-src/pep508_package/parent_folder/pep508-package/setup.py'), 
pyproject=PosixPath('/tmp/pipenv-fq_39y0k-src/pep508_package/pyproject.toml'), 
ireq=<InstallRequirement object: pep508_package from git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package editable=False>, 
extra_kwargs={'build_dir': '/tmp/reqlib-buildlhyan8qz', 'src_dir': '/tmp/pipenv-fq_39y0k-src/pep508_package', 
'download_dir': '/home/matteius/.cache/pipenv/pkgs', 'wheel_download_dir': '/home/matteius/.cache/pipenv/wheels'}, 
metadata=('name', 'parent-folder.sibling-package.src.sibling-package', 'version', '0.0.0', 'requires', ()), 
stack=<contextlib.ExitStack object at 0x7f2e0a77c580>, _finalizer=<finalize object at 0x7f2e0a7b2a00; for 'SetupInfo' at 0x7f2e0a703510>)

SETUP INFO BEFORE: SetupInfo(name='pep508_package', base_dir='/tmp/pipenv-3c06bgxi-src/pep508_package', _version='1.0.0', 
_requirements=frozenset({BaseRequirement(name='sibling-package', 
requirement=Requirement.parse('sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package')), 
BaseRequirement(name='six', requirement=Requirement.parse('six'))}), build_requires=('setuptools', 'wheel'),
 build_backend='setuptools.build_meta:__legacy__', setup_requires=(), python_requires=None, _extras_requirements=(), 
 setup_cfg=PosixPath('/tmp/pipenv-3c06bgxi-src/pep508_package/parent_folder/pep508-package/setup.cfg'), 
 setup_py=PosixPath('/tmp/pipenv-3c06bgxi-src/pep508_package/parent_folder/pep508-package/setup.py'), 
 pyproject=PosixPath('/tmp/pipenv-3c06bgxi-src/pep508_package/pyproject.toml'), 
 ireq=<InstallRequirement object: pep508_package from git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package editable=False>, 
 extra_kwargs={'build_dir': '/tmp/reqlib-buildfog79g70', 'src_dir': '/tmp/pipenv-3c06bgxi-src/pep508_package', 'download_dir': '/home/matteius/.cache/pipenv/pkgs', 
 'wheel_download_dir': '/home/matteius/.cache/pipenv/wheels'}, 
 metadata=('name', 'pep508-package', 'version', '1.0.0', 
 'requires', (Requirement.parse('six'), Requirement.parse('sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package'), Requirement.parse('six'), Requirement.parse('sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package')), ('dev', (Requirement.parse('parver'), Requirement.parse('invoke'), Requirement.parse('wheel'))), ('testing', (Requirement.parse('coverage'), Requirement.parse('flaky')))), 
 stack=<contextlib.ExitStack object at 0x7fb97e62b730>, _finalizer=<finalize object at 0x7fb97e6af200; for 'SetupInfo' at 0x7fb97e717f60>)

This is unfortunately taking me hours to debug, and will take hours more I am sure to try and unravel what is going on.

[abravalheri]

Hi @matteius, let me see if I can help you somehow.

You mentioned:

There is a code path that gets setup info, and what I see is in the working copy, we are getting in the metadata a valid version and requires attribute, but in the newer versions they are both empty in the metadata

I am assuming you are getting this info either from the (a) a generated wheel, (b) a generated sdist or (c) you run egg_info and inspect the generated metadata. Anyhow, is there a snippet of code that I could try to replicate this without using the internals of pipenv?

[abravalheri]

I wonder if it the metadata has empty version and requires because the name is all weird in the new version 'parent-folder.sibling-package.src.sibling-package'

This is comment intrigued me a little, because it might be related to another change in v61.*.

If you are trying to install parent-folder as if it was a package (note that parent-folder does not contain setup.py or pyproject.toml), previously setuptools would generate a UNKNOWN package, but now with the new name discovery it will indeed create parent-folder.sibling-package.src.sibling-package (the details of this mechanic were first proposed in https://github.com/pypa/setuptools/issues/2887#issuecomment-970708430)

This happens because parent_folder/sibling_package/src/sibling_package is accidentally a valid namespaced package under the eyes of PEP 420. For example, if you are inside the test-project, you can fire up the REPL and successfully run:

>>> import parent_folder.sibling_package.src.sibling_package

I have the impression that the name parent-folder.sibling-package.src.sibling-package is a valid under PEP 508.

[matteius]

Anyhow, is there a snippet of code that I could try to replicate this without using the internals of pipenv?

Unfortunately not yet and I may need to take a break from this for a while due to other obligations, I know the pip-tools project has a similar issue reported but no reproduction steps there either. Pipenv and pip pull in the latest setuptools like its candy, all over the place, so its really hard to restrict it back but I have done some of that and see some behavioral changes, but not enough to get the end result.

If you are trying to install parent-folder as if it was a package (note that parent-folder does not contain setup.py or pyproject.toml), previously setuptools would generate a UNKNOWN package, but now with the new name discovery it will indeed create parent-folder.sibling-package.src.sibling-package

The thing is I am using this url git+https://github.com/techalchemy/test-project.git@master#egg=pep508_package&subdirectory=parent_folder/pep508-package which calls out the subdirectory of the actual package name to use -- it should therefore not be trying to use parent_folder. This is valid under the URL Fragments section of the pip docs here: https://pip.pypa.io/en/stable/topics/vcs-support/#url-fragments

Pip is using setuptools under the hood for this, so I think that is where the problem enters, because pipenv vendors in pip, so I am deep in the internals of pip code and the resolvelib code trying to figure out how this changed without us changing anything.

[abravalheri]

On a side note: if pipenv/pip relies on the fact that projects may have name="UNKNOWN", that might no longer be true in v61.*, the new name discovery algorithm will try to always quick in.

[matteius]

On a side note: if pipenv/pip relies on the fact that projects may have name="UNKNOWN", that might no longer be true in v61.*, the new name discovery algorithm will try to always quick in.

We aren't directly relying on a name of UNKNOWN, I think when this actually works, it uses the right name from the #egg and the right subdirectory as well, yielding a package like pep508_package-1.0.0-py3-none-any.whl which matches the name from the Pipfile.

[matteius]

@abravalheri I have narrowed my test example down to a single factor, In requirementslib/models/setup_info.py if I modify the method build_pep517 to do this extra hack at the beginning (the hack has knowledge of the package sub-directories`:

def build_pep517(source_dir, build_dir, config_settings=None, dist_type="wheel"):
    if 'pep508_package' in source_dir or 'pep508-package' in source_dir:
        source_dir = f"{source_dir}/parent_folder/pep508-package"

When I have that extra logic disabled, and some extra print statements, I can see that this gets built: parent_folder.sibling_package.src.sibling_package-0.0.0-py3-none-any.whl

build pep517 base_dir /tmp/pipenv-ln7c16i9-src/pep508_package
pep517_config {'--global-option': []}
result of build_pep517: parent_folder.sibling_package.src.sibling_package-0.0.0-py3-none-any.whl

However, with my hack logic in place, the output instead is: pep508_package-1.0.0-py3-none-any.whl

build pep517 base_dir /tmp/pipenv-2xqe9gf8-src/pep508_package
pep517_config {'--global-option': []}
result of build_pep517: pep508_package-1.0.0-py3-none-any.whl

This makes all the difference in the world to get the expected result of:

[{"version": "1.16.0", "hashes": ["sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926", "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"], "name": "six", "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2'"}, {"subdirectory": "parent_folder/pep508-package", "ref": "4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648", "git": "https://github.com/techalchemy/test-project.git", "name": "pep508-package"}, {"subdirectory": "parent_folder/sibling_package", "ref": "4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648", "git": "https://github.com/techalchemy/test-project.git", "name": "sibling-package"}, {"version": "1.16.0", "hashes": ["sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926", "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"], "name": "six", "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'"}, {"subdirectory": "parent_folder/pep508-package", "ref": "4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648", "git": "https://github.com/techalchemy/test-project.git", "version": "", "name": "pep508-package"}, {"subdirectory": "parent_folder/sibling_package", "ref": "4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648", "git": "https://github.com/techalchemy/test-project.git", "version": "", "name": "sibling-package"}]

However, I am still head scratching about what could have changed to cause it to no longer automatically get the subdirectory on the path in this step, or why it would have changed behaviors ~4 days ago.

[abravalheri]

Hi @matteius this might be caused by the change we have been discussing previously...

Am I correct to assume that this function is being called with source_dir set as the parent directory instead of the directory that actually contains the package being build? If that is the case, why is it being called to build a directory that does not contain either setup.py or pyproject.toml?

Previous versions of setuptools would create an UNKOWN package. But now the name will be inferred to something different...

[matteius]

@abravalheri Thanks for being supportive and helping me asses what is going on. Quick question -- Is it possible that somehow now in the Metadata is now lacking the subdirectory where before it might have included something about it? We have this logic also in requirementslib that tries to pull the subdirectory from the requirement, but its always None ... It is hard to tell where setuptools begins and ends in some of this code, so I apologize for referencing things you may not be familiar with

    @classmethod
    def from_requirement(cls, requirement, finder=None):
        # type: (TRequirement, Optional[PackageFinder]) -> Optional[SetupInfo]
        ireq = requirement.as_ireq()
        subdir = getattr(requirement.req, "subdirectory", None)
        return cls.from_ireq(ireq, subdir=subdir, finder=finder)

[abravalheri]

Let me build the package normally and see what the generated metadata looks like.

[abravalheri]

These are all different ways of "building" the package and how the metadata looks like:

# Assuming the project is cloned somewhere
% cd test-project/parent_folder/pep508-package
% rm -rf src/pep508_package.egg-info build dist && pipx run build
% unzip -c dist/pep508_package-1.0.0-py3-none-any.whl pep508_package-1.0.0.dist-info/METADATA
...
Requires-Dist: sibling-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...
% tar xOf dist/pep508_package-1.0.0.tar.gz pep508_package-1.0.0/src/pep508_package.egg-info/requires.txt
six
sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...

% .venv/bin/python -m pip install -U setuptools
% rm -rf src/pep508_package.egg-info build dist && .venv/bin/python setup.py bdist_wheel
% unzip -c dist/pep508_package-1.0.0-py3-none-any.whl pep508_package-1.0.0.dist-info/METADATA
...
Requires-Dist: sibling-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...

% .venv/bin/python -m pip uninstall pep508-package
% .venv/bin/python -m pip install 'pep508-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/pep508-package'
% cat .venv/lib/python3.8/site-packages/pep508_package-1.0.0.dist-info/METADATA
...
Requires-Dist: sibling-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...

It seems that Requires-Dist in the metadata always contain the directory.

[abravalheri]

Sorry, I forgot one line in the previous log:

.venv/bin/python -m pip install 'pep508-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/pep508-package'

I just updated the previous comment to include it.

[matteius]

@abravalheri Well thanks for your help, I now have a patch now for requirementslib that is generic enough for any subdirectory fragment, but I still don't understand how this just broke. Maybe before it wasn't passing the subdirectory properly either and setuptools was searching and finding the first setup.py in a subdirectory?

Check it out, this test passed: https://github.com/pypa/pipenv/runs/5712591302?check_suite_focus=true with this change: https://github.com/pypa/pipenv/pull/5017

[abravalheri]

Uummm... It is a possibility but I don't think setuptools is that smart 😅

[AndydeCleyre]

Hi, I'm popping in from jazzband/pip-tools#1603, where I'm having trouble pinning down the problem, which may or may not be the same thing discussed here.

EDIT: It looks like the below-folded example was creating an invalid setup.py

Bad Dockerfile stuff

Does any of the knowledge grown here explain why building this `Dockerfile` fails: ```Dockerfile FROM ubuntu:20.04 RUN apt-get update && apt-get -y install python3-venv RUN python3 -m venv myvenv RUN . ./myvenv/bin/activate && pip install wheel "pip-tools<6.5" "pip>=21.2,<22" RUN echo -e "from setuptools import setup\n\nsetup(name='apollo', install_requires=['conan==1.45.0'])" > setup.py RUN . ./myvenv/bin/activate && pip-compile ``` ```console $ buildah bud Dockerfile ... subprocess.CalledProcessError: Command '['/myvenv/bin/python3', '/myvenv/lib/python3.8/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/tmp/tmp5ifphqs3']' returned non-zero exit status 1. error building at STEP "RUN . ./myvenv/bin/activate && pip-compile": error while running runtime: exit status 1 ``` While this interactive shell with `podman` succeeds?: ```console $ podman run -it --rm docker://ubuntu:20.04 # apt-get update && apt-get -y install python3-venv # mkdir ~/myproj # cd ~/myproj # python3 -m venv venv # . ./venv/bin/activate # pip install wheel "pip-tools<6.5" "pip>=21.2,<22" # echo -e "from setuptools import setup\n\nsetup(name='apollo', install_requires=['conan==1.45.0'])" >setup.py # pip-compile ```

[AndydeCleyre]

The pip-tools daily scheduled CI began failing four tests 5 days ago, the day of setuptools v61.0.0 release.

These are:

• test_cli_compile.py::test_input_formats[setup.py]
• test_cli_compile.py::test_multiple_extras[setup.py-singular]
• test_cli_compile.py::test_one_extra[setup.py]
• test_cli_compile.py::test_multiple_extras[setup.py-comma-separated]

[abravalheri]

The pip-tools daily scheduled CI began failing four tests 5 days ago, the day of setuptools v61.0.0 release.

Hi @AndydeCleyre, I noticed that the fixtures seem to be creating distributions with no python modules.

When I compare METADATA_TEST_CASES for setup.cfg and setup.py I can see that setup.cfg has packages = find:, but setup.py lacks the equivalent packages=find_packages(). Is that on purpose?

---

Note that, prior to setuptools v61, these different configurations are not equivalent:

a) The example with setup.cfg would include sample_lib/__init__.py in the distribution b) The example with setup.py would be empty.

For the majority of use cases, the situation in (b) is accidental and unintended (maybe the same accident happened in your test case?).

With the improvements in v61, setuptools will try to auto-detect which files to include, but it will fail if the root directory has a confusing/equivocal layout.

The example generated by the fixtures seems to have extra folders dists and packages.

setuptools>=61 does not know exactly how to deal with these folders and therefore, halts the build process asking for the user to improve the configuration or use a src-layout.

This is the breaking change motivating the major bump from v60 to v61 and is described in the CHANGELOG.

The layouts recognized by setuptools for auto-discovery are described in https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#automatic-discovery.

Layouts that differ from the ones documented are asked to explicitly list packages and/or py_modules in their configuration.

These include "intentionally empty" packages and "meta-packages"[^1]. For those scenarios the users are asked to set packages=[].

[^1]: That only exists to specify dependencies.

[andy-maier]

A late but big "Thank You!!" for fixing this :-)

We had already removed installation via setup.py from the documentation of our project some time ago, and I'll also remove it now from our install test.