abravalheri
commented
3 weeks ago Hi @ehkristen , could you please provide a mimum reproducible example so that we can investigate the issue?
Open ehkristen opened 3 weeks ago
abravalheri
commented
3 weeks ago Hi @ehkristen , could you please provide a mimum reproducible example so that we can investigate the issue?
ehkristen
commented
3 weeks ago setuptools version Not sure -> using py2exe
Python version Python 3.11
OS Edition Windows 10 Enterprise Version 22H2 OS build 19045.5011 Experience Windows Feature Experience Pack 1000.19060.1000.0
Additional environment information No response
Description I have an existing Python project that defines five packages, three modules (four if you count main.py), and a folder with files that get used at the top level,
The relevant directory structure is as follows:
.../src/backend:
├── package_a
| ├── __init__.py
| └── package_a.py
├── files
| └── (csv files in this folder)
├── package_b
| ├── __init__.py
| └── package_b.py
├── package_c
| ├── __init__.py
| └── package_c.py
├── package_d
| ├── __init__.py
| └── package_d.py
├── package_e
| ├── __init__.py
| └── package_e.py
├── module_a.py
├── module_b.py
├── main.py
├── setup.py
└── module_c.pyI have a build.cmd file (with a _build.cmd file) which tries to install the modules/packages and then it is expected for it to import all the packages/ modules.
The setup.py is:
from setuptools import setup
import setuptools
import distutils.core import setup
import py2exe
import os
setuptool.setup(
name="<Name>",
author="<Name>",
author_email="<name>@<company>.com",
description="<description>",
long_description= long_description,
long_description_content_type="text/markdown",
packages= ['package_a', 'package_b', 'package_c', 'package_d', 'package_e'],
package_dir={"package_a":"package_a", "package_b":"package_b",
"package_c":"package_c", "package_d":"package_d",
"package_e":"package_e"}
py_modules=[ "module_a", "module_b", "module_c"],
data_files=[ <os.walk of the file directory location, gathers all the file directories> ],
python_requires=">=3.6",
)
setup(console=['main.py'])Expected behavior
With python <= 3.8, installation works fine and the imports described above work fine and the executable main.exe gets built (But note: I was using the auto discovery function, which worked perfectly fine. Auto discovery with python 3.11 with my flat layout errors and build gets terminated).
How to Reproduce With python >=3.11 installation fails with the following error:
error: Multiple top-level modules discovered in a flat-layout: ['package_a', 'package_b', 'package_c', 'package_d', 'package_e', 'files'].
Documentation herecontains the note:
If you are using auto-discovery with flat-layout, setuptools will refuse to create distribution archives with multiple top-level packages or modules.
This is done to prevent common errors such as accidentally publishing code not meant for distribution (e.g. maintenance-related scripts).
Users that purposefully want to create multi-package distributions are advised to use Custom discovery or the src-layout.
However, as of yet I have been unable to find any combination of setup.py options that allows the packages to be installed and all modules to be imported with a successful build. Either installation fails with the above error about the flat-layout, or the main.exe can't locate the packages/modules when ran. Please note that I am not using a MANIFEST.in nor .egg-info files (did not need them before upgrading python from 3.8 to 3.11). I tried adding these files, but it made no difference in terms of outcomes.
When I try to change the structure of the project, which is a less than desirable option, I am able to have the build go through, but the resulting executable main.exe is not able to locate the modules/packages and crashes. I filed this as a bug because the existing directory structure worked fine with python 3.8, but there seems to be no support for it in python 3.11 (mainly that the 3 listed options did not work). Could you provide an example of a working setuptools setup.py that works with a flat-layout?
Did I simply just not find the right combination of configuration to make this work? (I tried different options for numerous hours and even looked through the setuptools source code.)
Output See above
abravalheri
commented
3 weeks ago Umm... we probably need to further simplify this reproducer. As described in minimum reproducible example, we should remove anything that is not related to the investigation of the problem. This includes removing py2exe (since the suspicion is of a bug in setuptools not in py2exe). The provided setup.py script also contains some syntax errors, so I assume it was not tested before posting above and that it also needs some extra fixes.
There seems to be a conceptual error in your setup.py script: it is calling setup(...) twice.
The first time it includes packages=... and py_modules=..., and that is exactly what it needed to disable the aut-discovery. The second time however, it does not. That is the likely origin of the problem. Since the second call does not have any of the arguments, the auto-discovery is triggered.
In general it does not make sense to call the setup(...) function multiple times: it should be called only once per setup.py file. If you have multiple packages, then you somehow need to organise them to have multiple setup.py files. Anyway, if you want to disable auto-discovery you need to ensure that any call to setup(...) contains either packages=... or py_modules=... as indicated in the error message.
Also, please do not import distutils.core.setup instead use setuptools.setup.
Please find bellow a small example of how the example project would work once fixed the points discussed above and removed the unrelated py2exe. I am also adding a pyproject.toml file as recommended by the setuptools docs. This example is running in Linux, but it is very likely that an equivalent setup in Windows would work fine:
> docker run --rm -it python:3.11-bookworm /bin/bash
mkdir /tmp/proj && cd /tmp/proj
mkdir package_a package_b package_c package_d package_e
touch package_a/__init__.py package_a/package_a.py
touch package_b/__init__.py package_b/package_b.py
touch package_c/__init__.py package_c/package_c.py
touch package_d/__init__.py package_d/package_d.py
touch package_e/__init__.py package_e/package_e.py
touch module_a.py
touch module_b.py
touch module_c.py
touch main.py
cat <<EOF > setup.py
import setuptools
setuptools.setup(
name="package",
author="<Name>",
author_email="<name>@<company>.com",
description="<description>",
long_description="long_description",
long_description_content_type="text/markdown",
packages= ['package_a', 'package_b', 'package_c', 'package_d', 'package_e'],
package_dir={"package_a":"package_a", "package_b":"package_b",
"package_c":"package_c", "package_d":"package_d",
"package_e":"package_e"},
py_modules=[ "module_a", "module_b", "module_c"],
# data_files=[ <os.walk of the file directory location, gathers all the file directories> ],
python_requires=">=3.6",
)
EOF
cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools>=75.2"]
build-backend = "setuptools.build_meta"
EOF
apt update -y
apt install tree
tree
# .
# ├── main.py
# ├── module_a.py
# ├── module_b.py
# ├── module_c.py
# ├── package_a
# │ ├── __init__.py
# │ └── package_a.py
# ├── package_b
# │ ├── __init__.py
# │ └── package_b.py
# ├── package_c
# │ ├── __init__.py
# │ └── package_c.py
# ├── package_d
# │ ├── __init__.py
# │ └── package_d.py
# ├── package_e
# │ ├── __init__.py
# │ └── package_e.py
# ├── pyproject.toml
# └── setup.py
python -m pip install .
# ...
# Successfully built package
# Installing collected packages: package
# Successfully installed package-0.0.0Could you please try fixing the points discussed above? Let me know how it goes and feel free to close the issue if the changes suggested above are enough to fix the problems.
Otherwise, if you want more help with the investigation, please make sure to provide an updated version of the reproducer, reduced to the minimum (without dependencies not maintained by setuptools such as py2exe), and that clearly showcases the error. Please also make sure to follow the tips in https://stackoverflow.com/help/minimal-reproducible-example.
ehkristen
commented
3 weeks ago Hi, sorry for the confusion! the second call at the bottom is part of the second portion of my setup call and is used to signify that I want the executable to be set to main.py (I did not include the parts below that line).
The second call is using the distutils.core.setup
the second call looks like this:
setup(console =['main.py']
followed by code that zips the library in a folder that gets saved in the /dist folder that's generated.
Having both calls was not an issue with python 3.8
abravalheri
commented
3 weeks ago Hi @ehkristen , many things might be happening since you last tried with 3.8, including things "accidentally working", changes in the version of setuptools that is used, or deprecated functionality being removed. So, I recommend paying attention to the points I highlighted above.
Please make sure that you include packages=... and of py_modules=... in any calls to setup(...), and please don't use distutils.core.setup. I also don't recommend executing setup(...) more than once in the same script, this is very likely error prone, and setuptools is not really designed to support that.
Moreover it is a good idea to have a look at any deprecation warnings and try to tackle them.
If you don't plan to submit a new version of the reproducer, let's close this issue because we have exhausted the investigation for the given example.
As I showed in my previous comment, once you fix the setup.py script and follow the suggestions in the error message, the problems seem to go away.
ehkristen
commented
2 weeks ago Hi @abravalheri , so I reworked everything, utilizing py2exe.freeze, but I'm experiencing an issue where the zipped library does not include the base python modules that are called in the script? (Example pandas) I tried explicitly writing out the module in the options={ "includes": "pandas"} but when I tell it to include pandas, the hook_pandas from hooks.py errors saying that there's no module named packaging . I checked inside of this function to see what was happening, and it appears to import from a module called packaging? The only packaging module that I see in Python 311 is under site-packages/pkg_resources/_vendor/packaging while hook.py is under py2exe.
abravalheri
commented
2 weeks ago Hi @ehkristen, I will not be able to assist you with py2exe related questions.
If you have a minimal reproducer that only uses setuptools and PyPA tools like build or pip (without py2exe), I can try to have a look, but I do need a proper minimal reproducible example for that (so if you are interested please make sure to share a reproducer that follow the steps in https://stackoverflow.com/help/minimal-reproducible-example).
packaging is used by setuptools, but because of the bootstrap limitations of the Python packaging ecosystem, setuptools ships its own vendored version of packaging. However, it will always try to first use the version that is already available in the environment, and just rely on the vendored copy as last resort.
ehkristen
commented
2 weeks ago Hi @abravalheri , is it possible to use setuptools to create an executable? From my understanding it does not make the executable?
abravalheri
commented
2 weeks ago Setuptools allows specifying executable scripts to be installed alongside packages via entry-points: https://setuptools.pypa.io/en/latest/userguide/entry_point.html .
Other than that, setuptools does not natively support creating single file executables.
Other tools may use setuptools as an underlying step in this process, but that logic is not implemented in the setuptools repository.
I’m experiencing an issue with setuptools, specifically that my directory layout is upsetting the auto discovery feature (for my packages and modules). The error I get is shown above, which instructs me to do one of three listed options… Option 2 is not an option for me. I tried setting up custom discovery which did not work (it still complained about the layout and ignored the exclude call). I also tried explicitly listing out the packages and modules, but it seemingly still tries to use auto discovery regardless….
I had my script working setuptools (using setup.py) with Python 3.8, but this issue is now occurring after upgrading to 3.11
Originally posted by @ehkristen in https://github.com/pypa/setuptools/issues/4013#issuecomment-2436409234