search

pypa / setuptools

Official project repository for the Setuptools build system
https://pypi.org/project/setuptools/
MIT License
2.34k stars 1.14k forks source link

[BUG] editable install broken for multiple entries in `package_dir` #4294

Closed con-f-use closed 1 month ago

con-f-use commented 1 month ago

setuptools version

69.2.0

Python version

3.11.8

OS

NixOS 23.11

Description

Given the following setup.py:

#!/usr/bin/env python
# setup.py
import setuptools

setuptools.setup(
    name="somepackage",
    package_dir={"shared": "libs/shared", "app_one": "apps/app_one"},
    packages=setuptools.find_packages(where="libs") + setuptools.find_packages(where="apps"),
)

Installing a package with multiple entries in package_dir results in all but one entry not being importable. That is because the created .egg-link contains an invalid entry with a relative path:

$ cat venv/lib/python3.11/site-packages/somepackage.egg-link 
/tmp/edt_bug
.   <--- this is invalid

See the console output below for more information on packaging and file-structure. All the files contain a line print(f"hello from {__file__}"). This problem occurs with equivalent setup.cfg. This might be a bug in pip rather than setuptools, depending on what delivers the content for the egg-link.

Expected behavior

Importing modules works the same with editable installs and regular package installs.

How to Reproduce

  1. Create a virtual env and make sure wheel, setuptools and pip are installed, and activate the venv
  2. Create the file structure as listed below
  3. Install the resulting package with pip install --editable ./
  4. Observe that you cannot import app_one
  5. Install the package without --editable and observe that it can now be imported

Output

### FILE STRUCTURE
$ tree
├── apps
│   └── app_one
│       ├── app_one.py
│       └── __init__.py
├── libs
│   └── shared
│       ├── __init__.py
│       └── mymod.py
└── setup.py
### CONTENTS
$ cat setup.py
#!/usr/bin/env python
import setuptools

setuptools.setup(
    name="somepackage",
    package_dir={"shared": "libs/shared", "app_one": "apps/app_one"},
    packages=setuptools.find_packages(where="libs") + setuptools.find_packages(where="apps"),
)

$ cat apps/app_one/app_one.py 
print(f"hello from {__file__}")
## ENVIRONMENT
$ python -m venv venv && source venv/bin/activate && $ chmod +x setup.py

$ pip install -U wheel setuptools pip
Installing collected packages: wheel, setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 65.5.0
    Uninstalling setuptools-65.5.0:
      Successfully uninstalled setuptools-65.5.0
Successfully installed setuptools-69.2.0 wheel-0.43.0

Problem with editable installs

$ pip install -e ./
Installing collected packages: somepackage
  Running setup.py develop for somepackage
Successfully installed somepackage-0.0.0

$ python -c 'import app_one; print(dir(app_one))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'app_one'
[ble: exit 1]

$ cat venv/lib/python3.11/site-packages/somepackage.egg-link 
/tmp/edt_bug
.

Works with regular installs:

$ ./setup.py bdist_wheel
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/shared
copying libs/shared/mymod.py -> build/lib/shared
copying libs/shared/__init__.py -> build/lib/shared
creating build/lib/app_one
copying apps/app_one/app_one.py -> build/lib/app_one
copying apps/app_one/__init__.py -> build/lib/app_one
warning: build_py: byte-compiling is disabled, skipping.

/tmp/edt_bug/venv/lib/python3.11/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/shared
copying build/lib/shared/mymod.py -> build/bdist.linux-x86_64/wheel/shared
copying build/lib/shared/__init__.py -> build/bdist.linux-x86_64/wheel/shared
creating build/bdist.linux-x86_64/wheel/app_one
copying build/lib/app_one/app_one.py -> build/bdist.linux-x86_64/wheel/app_one
copying build/lib/app_one/__init__.py -> build/bdist.linux-x86_64/wheel/app_one
warning: install_lib: byte-compiling is disabled, skipping.

running install_egg_info
running egg_info
writing somepackage.egg-info/PKG-INFO
writing dependency_links to somepackage.egg-info/dependency_links.txt
writing top-level names to somepackage.egg-info/top_level.txt
reading manifest file 'somepackage.egg-info/SOURCES.txt'
writing manifest file 'somepackage.egg-info/SOURCES.txt'
Copying somepackage.egg-info to build/bdist.linux-x86_64/wheel/somepackage-0.0.0-py3.11.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/somepackage-0.0.0.dist-info/WHEEL
creating 'dist/somepackage-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'app_one/__init__.py'
adding 'app_one/app_one.py'
adding 'shared/__init__.py'
adding 'shared/mymod.py'
adding 'somepackage-0.0.0.dist-info/METADATA'
adding 'somepackage-0.0.0.dist-info/WHEEL'
adding 'somepackage-0.0.0.dist-info/top_level.txt'
adding 'somepackage-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel

$ pip install dist/somepackage-0.0.0-py3-none-any.whl
Looking in indexes: http://10.17.65.203/root/devel/+simple/
Processing ./dist/somepackage-0.0.0-py3-none-any.whl
Installing collected packages: somepackage
  Attempting uninstall: somepackage
    Found existing installation: somepackage 0.0.0
    Uninstalling somepackage-0.0.0:
      Successfully uninstalled somepackage-0.0.0
Successfully installed somepackage-0.0.0

$ python -c 'import app_one; print(dir(app_one))'
hello from /tmp/edt_bug/venv/lib/python3.11/site-packages/app_one/__init__.py
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
abravalheri commented 1 month ago

Hi @con-f-use, please note that calling setuptools via python setup.py <...> is a deprecated method, and that is what pip seems to be doing (in the logs you can see Running setup.py develop for somepackage). That is probably because this package is triggering pip's legacy behaviour (due to the absence of pyproject.toml).

pythont setup.py develop has a series of well know limitations (e.g. it cannot support package_dir properly), and it is unlikely we are going to improve on those (specially because it is already deprecated, so it is offered as it is and it will be removed in the future).

Now you can try to add a pyproject.toml or call pip using the --use-pep517 flag to trigger the "new behaviour"[^1] and see if that works for you. Please note however that this "new behaviour" is not fully equivalent/backward compatible to python setup.py <...>, and the original source code might need to be adapted. (The special implication is the "build isolation" as described in pip's docs and the need to define the build dependencies in pyproject.toml).

Note however that the "new" editable installation method also have limitations, see https://setuptools.pypa.io/en/latest/userguide/development_mode.html#limitations. If you browse the discussions in the Python Packaging Discourse, you can probably find long threads about editable installs and how there is no bullet proof solution for all use cases and that so far it is impossible to be 100% with a "final installation" while allowing users to change the implementation dynamically.

[^1]: Note that this behaviour is actually several years old already and it was initially defined in PEP 517 (2015). I am calling it "new behaviour" for lack of better words.