search

pypa / build

A simple, correct Python build frontend
https://build.pypa.io
MIT License
697 stars 115 forks source link

How to specify packages in pyproject.toml - build fails #777

Closed andy-maier closed 2 months ago

andy-maier commented 2 months ago

I have a project with two Python packages. It uses the flat layout and has a pyproject.toml file (see below) that uses setuptools as the backend, and its setup.py file has just a call to setuptools.setup() without any arguments.

When I invoke "build", it fails because it cannot identify the Python packages:

$ python -m build
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - setuptools-scm>=8.0
  - setuptools>=61.0
  - wheel>=0.43.0
* Getting build dependencies for sdist...
error: Multiple top-level packages discovered in a flat-layout: ['try', 'done', 'perf', 'attic', 'design', 'images', 'pywbem', 'issues', 'packaging', 'tmp_issues', 'test_pylint', 'pywbem_mock'].

To avoid accidental inclusion of unwanted files or directories,
setuptools will not proceed with this build.

If you are trying to create a single distribution with multiple packages
on purpose, you should not rely on automatic discovery.
Instead, consider the following options:

1. set up custom discovery (`find` directive with `include` or `exclude`)
2. use a `src-layout`
3. explicitly set `py_modules` or `packages` with a list of names

To find more information, look for "package discovery" on setuptools docs.

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist

When I try to follow the option 3, it fails as well, saying that packages is not allowed (the same happens with py_modules):

$ python -m build
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - setuptools-scm>=8.0
  - setuptools>=61.0
  - wheel>=0.43.0
* Getting build dependencies for sdist...
Traceback (most recent call last):
  File "/Users/maiera/virtualenvs/pywbem312/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
    main()
  File "/Users/maiera/virtualenvs/pywbem312/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maiera/virtualenvs/pywbem312/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 287, in get_requires_for_build_sdist
    return hook(config_settings)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/build_meta.py", line 328, in get_requires_for_build_sdist
    return self._get_build_requires(config_settings, requirements=[])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
    self.run_setup()
  File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup
    exec(code, locals())
  File "<string>", line 360, in <module>
  File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/__init__.py", line 104, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 158, in setup
    dist.parse_config_files()
  File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/dist.py", line 631, in parse_config_files
    pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
  File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py", line 68, in apply_configuration
    config = read_configuration(filepath, True, ignore_option_errors, dist)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py", line 129, in read_configuration
    validate(subset, filepath)
  File "/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/build-env-vw8dmmgw/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py", line 57, in validate
    raise ValueError(f"{error}\n{summary}") from None
ValueError: invalid pyproject.toml config: `project`.
configuration error: `project` must not contain {'packages'} properties

The pyproject.toml file with packages specified looks like this:

[build-system]
requires = [
    "setuptools>=61.0",
    "setuptools-scm>=8.0",
    "wheel>=0.43.0",
]
build-backend = "setuptools.build_meta"

[project]
name = "pywbem"
packages = [
    "pywbem",
    "pywbem_mock"
]
description = "pywbem - A WBEM client"
authors = [
    {name = "Tim Potter", email = "tpot@hp.com"}
]
maintainers = [
    {name = "Andreas Maier", email = "maiera@de.ibm.com"},
    {name = "Karl Schopmeyer", email = "k.schopmeyer@swbell.net"}
]
readme = "README.md"
license = {text = "LGPL version 2.1, or (at your option) any later version"}
keywords = ["cim", "wbem", "client"]
classifiers = [
    "License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+)",
    "Development Status :: 5 - Production/Stable",
    "Intended Audience :: Developers",
    "Intended Audience :: System Administrators",
    "Topic :: Software Development :: Libraries :: Python Modules",
    "Topic :: System :: Systems Administration",
    "Environment :: Console",
    "Operating System :: OS Independent",
    "Programming Language :: Python :: 2",
    "Programming Language :: Python :: 2.7",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.6",
    "Programming Language :: Python :: 3.7",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
]
requires-python = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*"
dynamic = ["version"]
dependencies = [

    # Direct dependencies for install (must be consistent with minimum-constraints-install.txt)

    "mock>=2.0.0,<4.0.0; python_version == '2.7'",
    "ply>=3.10",
    # PyYAML 5.3 fixed narrow build error on Python 2.7
    # PyYAML 5.3.1 addressed issue 38100 reported by safety
    # PyYAML 5.2 addressed issue 38639 reported by safety
    # PyYAML 6.0 removed support for Python 2.7, but 6.0b1 installs in installtest
    "PyYAML>=5.3.1,<6.0; python_version == '2.7'",
    "PyYAML>=5.3.1; python_version >= '3.6'",
    # virtualenv 20.0.0 (required on py3.8+) requires six>=0.12.0
    # tox 3.17 (used in dev-requirements.txt) requires six>=1.14.0
    # six 1.16.0 removes the ImportWarning raised by Python 3.10
    "six>=1.14.0; python_version <= '3.9'",
    "six>=1.16.0; python_version >= '3.10'",
    # requests 2.22.0 removed the pinning of urllib3 to <1.25.0, and urllib 1.25.9
    #   is required to address safety issues
    # requests 2.25.0 tolerates urllib3 1.26.5 which is needed on Python 3.10 to
    #   remove ImportWarning in six
    "requests>=2.25.0; python_version == '2.7'",
    "requests>=2.25.0; python_version == '3.6'",
    "requests>=2.31.0; python_version >= '3.7'",
    # yamlloader 1.1.0 gets istalled by mistake on py27+34 on Ubuntu when using setup.py install. See issue #2745.
    "yamlloader>=0.5.5,<1.0.0; python_version == '2.7'",
    "yamlloader>=0.5.5; python_version >= '3.6'",

    # Indirect dependencies for install that are needed for some reason (must be consistent with minimum-constraints-install.txt)

    # setuptools 61.0.0 breaks "setup.py install", see https://github.com/pypa/setuptools/issues/3198
    "setuptools!=61.0.0; python_version >= '3.7'",

    # urllib3 1.24.1 addressed issue 37055 reported by safety
    # urllib3 1.24.2 addressed issue 37071 reported by safety
    # urllib3 1.25.9 addressed issue 38834 reported by safety
    # urllib3 needs to be pinned to <1.25 for requests <2.22.0
    # urllib3 1.26.5 vendors six 1.16.0 which is needed on Python 3.10 to remove ImportWarning
    # urllib3 2.0 requires py>=3.7
    "urllib3>=1.26.18,<2.0.0; python_version == '2.7'",
    "urllib3>=1.26.18,<2.0.0; python_version == '3.6'",
    "urllib3>=1.26.18; python_version >= '3.7'",

    # certifi 2020.6.20 removed support for Python 2.7 but does not declare it
    "certifi>=2019.11.28,<2020.6.20; python_version == '2.7'",
    "certifi>=2023.07.22; python_version >= '3.6'",

    "idna>=2.8",

]

[project.optional-dependencies]
test = ["TBD"]

[project.urls]
Homepage = "https://pywbem.github.io/pywbem/"
"Bug Tracker" = "https://github.com/pywbem/pywbem/issues"
Documentation = "https://pywbem.readthedocs.io/en/latest/"
Repository = "https://github.com/pywbem/pywbem"
Changelog = "https://pywbem.readthedocs.io/en/latest/changes.html"

[tool.setuptools_scm]
# Get the version from the Git tag, and write a version file:
version_file = "pywbem/_version.py"

This happened with: macOS Python 3.12 build 1.2.1 setuptools 69.5.1 wheel 0.43.0

My questions are:

abravalheri commented 2 months ago

You probably are looking for tool.setuptools.packages or tool.setuptools.py-modules. Ref: https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#setuptools-specific-configuration (or https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#custom-discovery if you have a nested tree of packages and don't want to specify one by one).

andy-maier commented 2 months ago

Yes, that's what I was looking for. Thanks much! My question 1 is addressed with that.

On question 2: Any chance something can be done about the tracebacks?

layday commented 2 months ago

The error occurs in a subprocess and arrives as a subprocess.CalledProcessError (with its output already having been emitted), which is next to useless. Unfortunately, this is intrinsic to how build hooks are invoked. A configuration error class would have to be standardised and configuration errors captured by the hook caller before build would be able to hide the stack trace.

layday commented 2 months ago

If you feel like this is something worth pursuing, you can start a discussion at https://discuss.python.org/c/packaging/14. I'll close this now.